The hype around multi-agent LLM systems often ignores a critical security flaw: data doesn't just stay in the final output. A new open-source project, AgentLeak, has launched on GitHub to benchmark privacy leakage across the hidden channels of agent orchestration. Accompanying an IEEE Access paper by El Yagoubi, Badu-Marfo, and Al Mallah, this tool moves beyond simple output checks to audit the messy internals of how agents talk to each other and their tools.

The Internal Channel Crisis

The core finding is unsettling for anyone deploying autonomous agents in regulated industries. While final outputs (C1) are often sanitized, internal channels (C2) like inter-agent messages and shared memory (C5) leak sensitive information at a rate of 74.0% on average. This is 2.6 times higher than external leaks. The study evaluated 1,000 scenarios across healthcare, finance, legal, and corporate domains, utilizing 4,979 validated execution traces across five major models including GPT-4o, Claude-3.5-Sonnet, and Llama-3.3-70B.

The Claude Paradox and Tool Leaks

Claude-3.5-Sonnet presents a fascinating paradox: it has the lowest final output leakage at 8.2%, yet exhibits the highest internal-to-external ratio at 6.6x. This means while the user sees a clean answer, the internal chatter is filled with raw, unmasked data. Furthermore, 'Finding 7' highlights that tool inputs (C3) and system logs (C6) are massive vectors for leakage, with rates hitting up to 85% on Claude 3.5, even when the final response is perfectly sanitized. If you are logging tool calls for debugging, you are likely logging your users' secrets.

Practical Tooling and Reproduction

For builders who need more than academic stats, AgentLeak offers a practical Python SDK and CLI available via pip install agentleak. The tool supports local trace analysis without LLM dependencies, allowing for deterministic checks on sensitive data in tool calls, logs, and generated files. It integrates with popular frameworks like CrewAI, LangChain, AutoGPT, and MetaGPT. Developers can run a quick synthetic healthcare example with agentleak run --scenario healthcare_patient_summary to see these leaks in action immediately.

Key Takeaways

  • Internal channels (inter-agent messages, memory) leak 2.6x more data than final outputs.
  • Standard output-only audits miss nearly 46% of privacy violations.
  • Tool inputs and logs are high-risk vectors, with leakage rates up to 85%.
  • The project provides both a research benchmark and a practical SDK for CI integration.

The Bottom Line

If you aren't auditing your agent's internal monologue and tool logs, you aren't auditing your privacy compliance. AgentLeak proves that the 'black box' isn't just the modelβ€”it's the entire orchestration layer.

Sources

  • Hacker News: [AgentLeak GitHub Repository](https://github.com/Privatris/AgentLeak)

> From The Wire