In a recent stress test of the Glama.ai MCP catalog, developer mrsergeiluzhnov discovered that a budget-friendly AI agent, guided by the Xonta orchestrator, significantly outperformed a premium model in identifying destructive tools. The experiment analyzed 998 remote servers and 19,944 tools, highlighting a critical inefficiency in current LLM-based agent architectures: bigger models are not necessarily safer models. By leveraging publisher-provided readOnlyHint and destructiveHint metadata, the test demonstrated that cost-aware orchestration can reduce the risk of irreversible actions while slashing token costs by 2.5 times.

The Flaw in Name-Based Scanning

The first phase of the study compared two methods of tool classification: a simple scanner that reads only tool names versus a GigaChat agent that analyzes full descriptions. The results were damning for the scanner. Because many irreversible actions use benign names like save_item or resolve_task, the scanner frequently misclassified dangerous writes as harmless reads. Specifically, the scanner took a quarter of irreversible tools for harmless reads. Conversely, the agent rarely mistook dangerous tools for reads (only 2 out of 100 errors), but it incorrectly labeled half of the irreversible tools as reversible. The takeaway for infrastructure builders is clear: relying on tool names alone is insufficient for safety-critical agents.

Orchestrator Bugs and Harm-Cost Logic

The testing process exposed significant flaws in Xonta version 0.21. The orchestrator initially failed to recognize that certain optimization tricks reduced overall harm despite slightly lowering accuracy. A bug in the scoring logic treated all errors equally, ignoring the fact that a false negative on a delete_payment tool costs far more than a false positive on a read_config tool. This led to the release of Xonta 0.22, which introduced a harm-cost metric and a new outcome category for same quality, just cheaper, allowing the system to validate optimizations that trade minor accuracy for major safety and cost savings.

Premium Models Can Be More Dangerous

Perhaps the most counterintuitive finding emerged in Stage Three, where the team introduced GigaChat-2-Max, a significantly larger and more expensive model. While this premium model achieved the highest overall accuracy on the test set, it proved to be the most dangerous in practice. It frequently underestimated the irreversibility of actions, leading to more high-cost mistakes. The cheap agent, equipped with the orchestrator’s stricter-of-two heuristic—which compares the agent’s judgment against a conservative scanner—caught dangerous tools as effectively as the premium model but at a fraction of the cost.

Key Takeaways

  • Publisher metadata (hints) is a vital but underutilized signal for agent safety, covering 95.9% of tools in the Glama catalog.
  • A cheap model with a well-designed orchestrator can outperform expensive models by minimizing harm cost rather than maximizing raw accuracy.
  • Agent self-reporting is unreliable; independent verification showed that every fourth or fifth all good assessments failed re-checks.
  • Prompt structure is fragile; a single field naming conflict caused a third of model responses to fail parsing in early runs.
  • Xonta 0.22’s harm-cost scoring is essential for validating optimizations that trade accuracy for safety.

The Bottom Line

Stop paying for premium tokens to do basic safety checks. If your agent architecture doesn't account for the asymmetric cost of errors, you're just burning money while increasing risk.