Every call to a large language model drains your budget and your patience. You pay for every input and output token, then wait while the model processes your prompt. In high-traffic applications, a significant portion of these calls are redundant repeats of the same help-center questions, system prompts, or documents. AI caching is the practice of identifying that redundancy and avoiding the cost of processing it twice.

Why AI Caching Is Not Just Web Caching

Classic web caching works because requests for static assets like /logo.png always return identical bytes. AI requests are messier. Outputs are non-deterministic, meaning the same question can yield different wordings. Inputs are large and mostly repeated, with only a few dozen new tokens at the end of a chat request. Furthermore, semantic equivalence is hard to detect; 'How do I reset my password?' and 'password reset steps?' mean the same thing but share almost no characters. This complexity necessitates a layered approach rather than a single technique.

Layer 1 and 2: Exact and Semantic Response Caching

Exact response caching stores the complete output of a request. If an identical request arrives, the system returns the stored response without contacting the model. In ZRouter, an open-source OpenAI-compatible gateway, this is backed by Redis and includes canonicalization to normalize JSON key order and whitespace. It also uses request coalescing, ensuring that if ten identical requests arrive simultaneously, only one hits the provider while the others wait for the result. Semantic caching takes this further by using embeddings to find similar prompts. ZRouter compares the vector similarity of the final user message against stored prompts, returning cached answers if the cosine similarity exceeds a threshold, such as 0.92.

Layer 3 and 4: Provider and Embedding Caching

Provider prompt caching is distinct because it does not reuse the answer. Instead, it reuses the model’s work of processing a repeated prefix, such as a long system prompt or conversation history. Providers like OpenAI, Anthropic, and Google Gemini implement this differently, often requiring specific cache keys or breakpoints. ZRouter automates this via a provider prompt-cache planner that applies the correct directive for each service. Embedding caching stores vectors for text snippets to avoid re-embedding identical content, which is crucial for RAG pipelines. ZRouter integrates this into its exact cache for /v1/embeddings and uses in-memory reuse for identical semantic lookups within a five-minute window.

Key Takeaways

  • AI caching is a stack of four layers: exact response, semantic response, provider prompt, and embedding.
  • Exact and semantic caches skip the model entirely but return a potentially stale answer.
  • Provider prompt caching keeps answers fresh but reduces input costs and latency for repeated prefixes.
  • ZRouter implements all layers in one gateway, using Redis and vector stores like Qdrant or Pinecone.

The Bottom Line

Stop treating AI costs as fixed. By stacking these caching layers, you can turn your LLM bill from a runaway train into a manageable, predictable expense.