As AI agents move from novelty to infrastructure, the cost visibility gap is widening. Developer Josh Hall recently detailed his workflow for tracking token spend across three distinct AI interfaces: a Claude coding assistant, a Codex CLI for pair-programming, and a local Ollama instance handling overnight alert triage. The problem wasn't lack of usage; it was the opacity of the bill.
The Homelab Cost Blind Spot
Hallβs setup relies on two subscription-based services where quota burn is invisible in real-time, alongside a self-hosted model where compute costs are implicit. Without unified telemetry, he couldn't answer the fundamental question: what is the actual marginal cost of his AI-assisted development loop? This is a common failure mode for engineers treating LLMs as black-box utilities rather than observable services.
Unifying Telemetry with Grafana
The solution leverages Grafana, the open-source observability platform, to aggregate token metrics from disparate sources. By instrumenting the Claude and Codex APIs to emit usage data, and querying Ollamaβs local stats endpoint, Hall created a single pane of glass. This approach treats token consumption like any other system metricβCPU, memory, or network I/Oβallowing for alerting, historical analysis, and precise cost attribution per agent.
Key Takeaways
- Subscription AI tools obscure marginal cost; local models obscure compute cost. Both require explicit instrumentation.
- Grafana serves as an effective aggregation layer for heterogeneous AI telemetry, bridging cloud APIs and local runtimes.
- Overnight autonomous agents, like Hallβs alert triage system, are often the highest-cost invisible components in a dev stack.
The Bottom Line
If you aren't logging tokens, you aren't engineering; you're gambling with a credit card. Observability is the only way to turn AI from a sunk cost into a managed service.