Sampact LLC has launched AgentProbe, a testing framework that abandons static test scripts in favor of dynamic chaos engineering for AI agents. The tool runs your agent in a live environment, then deliberately injects failures into the surrounding system to observe how the agent recovers. Unlike traditional unit tests that check for happy paths, AgentProbe focuses on the messy reality of production: tool errors, stale data reads, and prompt injections that break agent logic mid-execution.

Adversarial Testing Over Scripted Checks

The core innovation here is the 'adaptive Injector.' Instead of hardcoding a failure at step three, a second AI monitors the agent's real-time trajectory and chooses the precise moment to strike based on current context. This mimics the unpredictability of real-world infrastructure, where a database timeout or an ambiguous API response can happen at any point. The system checks for fourteen specific failure modes, including 'phantom_success' (where a commit works but the response is lost) and 'social_pressure' (where an agent bends policy due to urgent customer language).

Low-Barrier Entry for Developers

AgentProbe requires no code to set up the initial scenario. Developers describe business rules in plain English, such as refund windows or approval requirements, and point the tool at their agent's HTTP endpoint. The test runs using the developer's own Anthropic API key, ensuring that the cost of the adversarial injector is billed directly to the user rather than hidden in a SaaS subscription. This transparency extends to the final report, which tracks token usage down to the cache pricing, a detail most monitoring tools ignore.

Caveats and Open Source Roots

The team is upfront about the limitations: the test scenarios are generated by AI, not human QA. They advise users to skim the generated logic to ensure the rules are interpreted correctly, as a subtly wrong test can pass without proving anything. For those hesitant to spend API credits immediately, Sampact offers an open-source library mode that makes zero API calls, allowing developers to observe behavior locally before committing to a full run. It is a pragmatic approach to validating agent reliability without drowning in manual edge-case creation.

Key Takeaways

  • AgentProbe uses a second AI to dynamically inject failures into live agent runs rather than relying on static scripts.
  • The tool tests for 14 specific failure modes, including stale reads, prompt injections, and social pressure scenarios.
  • Setup requires only a plain English description of business rules and the developer's own Anthropic API key.
  • An open-source library mode allows for zero-cost local testing before using paid API credits.

The Bottom Line

If your agent can't handle a tool timeout gracefully, you don't have a product, you have a liability. AgentProbe finally forces developers to confront the fragility of their LLM logic before customers do. This is a necessary evolution in agent tooling. We are moving past 'does it answer the prompt?' to 'does it survive reality?'