The days of manually typing three messages into a chat window to test an AI agent are over. If your team relies on that 'looks good' vibe check, you are one Friday afternoon prompt edit away from a Monday morning disaster. Tamesu, an open-source CLI written in Go, is here to formalize the chaos. It introduces a structured testing framework where simulated users interact with your agent, and LLM judges grade the outcomes, all running directly in your CI pipeline.

The Architecture of Deterministic Chaos

Tamesu operates on a three-part test structure that eliminates the 'it worked on my machine' excuse. First, it deploys a persona with a specific scenarioβ€”a simulated user with a goal and a mood, like Ken, who is insistent and unhappy. Second, it targets your agent wherever it runs, supporting REST, gRPC, and even LiveKit rooms. Third, it enforces assertions. These aren't just string matches; they include deterministic checks on tool calls, backend state verification, latency metrics, and semantic grading via LLM judges. The tool is designed to catch what human eyes miss. In the provided example, a 'Mochi Shoes' support agent uses Genkit Go and Gemini. The agent has a rule to refuse refunds after 30 days, enforced only by the prompt. When a product manager softens the prompt to 'customer happiness comes first,' the agent starts refunding orders from 40 days ago. Tamesu catches this regression instantly. The judge fails the semantic check, the tool_not_called assertion flags that issue_refund was called when it shouldn't be, and the HTTP check confirms the backend status changed to 'refunded' instead of 'delivered.'

CI Integration and Baseline Comparison

This isn't just a local dev tool; it’s built for the pipeline. By saving a 'baseline' run, Tamesu can compare subsequent runs against known good states. The output is precise: it flags new failures, fixed issues, and performance regressions. For instance, a prompt change might not break logic but could spike latency. Tamesu’s metric assertions caught a 42% increase in turn latency in the example suite, blocking the merge automatically. The tool also generates JUnit and HTML reports, ensuring that visibility is baked into the development workflow. The configuration is decoupled from the agent’s code. You don’t need an SDK; if your agent speaks HTTP and JSON, Tamesu can test it. The target definition uses JSONPath to extract replies and tool calls, creating a contract between the tester and the system. This allows for rigorous testing of complex interactions, such as verifying that an agent calls lookup_order before issue_refund, or that it transfers to a human within one turn when requested.

Key Takeaways

  • Tamesu replaces manual chat testing with automated, repeatable scenarios using simulated users with distinct personas and goals.
  • The framework uses LLM judges for semantic grading and deterministic checks for tool calls, backend state, and latency.
  • Baseline comparisons in CI automatically block merges when prompt edits cause functional or performance regressions.
  • The tool supports multiple protocols (REST, gRPC) and requires no SDK integration into the agent under test.
  • Security and compliance are addressed through red-team playbooks mapped to the OWASP LLM Top 10.

The Bottom Line

If you aren't testing your agent's prompt changes with the same rigor as your code, you're gambling with your brand. Tamesu turns vibes into verifiable data, and that is a win for engineering sanity.