Standard observability tools are lying to you by omission. A recent deep dive from The Agent Loop on DEV.to exposes a critical blind spot in AI agent deployment: while dashboards accurately track token consumption, they completely fail to account for the massive overhead of crash recovery. The author, running an agent inside a Linux container on a phone, discovered that a third of all message traffic was dedicated to recovering from failures rather than executing useful tasks. This isn't just a quirk of unstable hardware; it's a systemic failure in how we measure agent efficiency.

The Illusion of Token Efficiency

The FinOps Foundation’s April 2026 guide on AI tooling costs explicitly names 'error rates and retry rates' as key metrics for visibility, labeling repeated calls due to errors as 'pure waste.' Yet, as the author points out, every major cost tool focuses on token counts because they are machine-readable. Outcomes, however, require human judgment to define 'done.' Consequently, the metric that actually matters—cost per use case—is ignored because naming it requires admitting which runs produced nothing. A crashed session still burns tokens, and if your dashboard only sees tokens, it sees activity where there is only failure.

Data from the Trenches

The author’s analysis of 19,090 message parts across 64 sessions over 56 days revealed a heavy tail distribution. While the median session contained just 18 messages, the largest stretched to 1,327. When categorized by intent, crash recovery sessions averaged 164 messages each, an order of magnitude higher than the median. In stark contrast, sessions that actually shipped work averaged significantly lower message counts. Twenty-four posts were produced from the 11% of traffic labeled 'ship work,' while the rest bought system stability. The caveat here is important: the author ran this on a phone inside a Linux container, a fragile setup, but the principle holds that recovery costs are disproportionately high.

Defining the Denominator

To fix this, you cannot rely on existing spend dashboards. You must build your own denominator. The author suggests a three-step process: first, name the outcome so a machine can verify it (e.g., 'a file exists at this path'); second, count both spend and outcomes; and third, let the ratio embarrass you. If you run 200 times to ship 4 items, your cost per outcome is fifty times your cost per run. This figure, not the raw token count, should drive your architecture decisions. The author’s own definition of 'done' evolved from simply publishing a post to including a verification step that no errors were found, highlighting that the definition itself must be tuned to survive contact with reality.

Key Takeaways

  • Standard agent dashboards track tokens but miss the 'absence' of work caused by retries and crashes.
  • Crash recovery sessions can consume an order of magnitude more messages than successful sessions.
  • Defining 'done' is a judgment call, but it is the only way to calculate true cost per outcome.
  • A wrong heuristic is better than no heuristic; start small and manually verify the definition of success.

The Bottom Line

Stop trusting your token counts. If you aren't manually verifying what constitutes a 'success' in your agent's output, your dashboard is just an expensive receipt for failure.