A developer on DEV.to has documented an illuminating journey into AI cost optimization that ended with a counterintuitive finding: the sophisticated routing system they built barely moved the needle compared to fixing how prompts were being cached.

The Router That Wasn't the Problem

The author, who goes by frankchu on DEV.to, has written extensively about model routing strategies. Their core thesis has consistently been that coding models have converged in capability, most agent work is mechanical in nature, and the pragmatic approach is to run cheap models by default while reserving expensive frontier models only for tasks that genuinely require them. They even referenced NVIDIA's efforts to productize this tiered approach. Armed with this philosophy, they set out to build a router that would intelligently dispatch coding tasks to appropriate model tiers based on complexity estimates. The expectation was clear: significant cost savings through smart routing alone.

When Caching Became the Villain

What they discovered instead was that prompt cachingβ€”the mechanism designed to reduce costs by reusing context from previous requestsβ€”had become their primary expense driver. The repeated token patterns in agentic coding workflows were triggering cache misses at surprising rates, causing the system to pay full price for requests that should have been cheap. "I spent all this time building routing logic," the author noted in their post, "and prompt caching was the whole problem." The router they built became almost irrelevant once they understood where the actual money was going.

Why This Matters for LLM Deployments

This finding highlights a fundamental tension in how we think about AI cost optimization. Most tooling discussions focus on model selection and routing logic, but caching behavior can quietly dominate the economics of sustained deployments. For teams running coding agents at scale, understanding cache hit rates across different prompt structures may be more valuable than any routing algorithm.

Key Takeaways

  • Prompt caching behavior varies dramatically based on prompt structure and repetition patterns
  • Model routing provides diminishing returns if underlying token efficiency isn't addressed first
  • Agentic workflows with repeated scaffolding generate cache-miss patterns that negate savings
  • Teams should audit actual cache hit rates before investing in complex routing infrastructure

The Bottom Line

The industry has been so focused on model prices and context windows that we've overlooked a simpler truth: how you structure prompts matters more than which model you pick. Before building your next elaborate routing system, check if your caching is actually working.