Before you wire up a search index, a retriever, and a chat model, do one piece of planning that will save you rework later: estimate your token budget for each query. In retrieval-augmented generation (RAG) apps, the model does not just read the user's question. It also reads your instructions, the retrieved passages, and any conversation history, and it still needs room to write an answer.

The Hidden Cost of Context

If those pieces add up to more than the model's context window, something gets cut off, and it is usually the answer or the most useful passage. Start with a simple per-query checklist to identify where your tokens are going. System instructions, including rules, tone, and citation guidance, often consume more than developers expect. The user question is usually small, but longer in follow-up turns. Retrieved chunks are often the largest part of the prompt, so multiply the number of chunks you retrieve by their average size.

Calculating the Real Number

You must also account for conversation history if you actually send past turns to the model, and reserve specific space for a complete answer, not just a few sentences. Then add a safety buffer. Real documents are uneven, metadata adds tokens, and a question that looks short can retrieve a long passage. A buffer of around 10 to 20 percent is a practical starting point to prevent edge-case failures.

Worked Example: The 99 Million Token Problem

A worked example makes the trade-off clear. Suppose your instructions take 800 tokens, the question takes 150, and you retrieve five chunks of 400 tokens each. That is 2,950 input tokens before the answer. If you reserve 350 tokens for output, each query uses 3,300 tokens. At 1,000 queries a day for 30 days, that is 99 million tokens before any buffer. Change the chunk count or size, and the monthly total moves quickly.

Build With Numbers, Not Hope

That is why the budget should come before the build. Once you know how much room retrieval can safely use, you can choose a sensible number of chunks, a chunk size that fits the window, and a model whose context limit leaves room for output. If the numbers do not fit, trim retrieval, shorten instructions, or summarize history, rather than discovering truncation in production. For a quick check of your own numbers, try the RAG Token Budget Calculator from AI Calculators to compare per-query totals with your model's limit.

Key Takeaways

  • Calculate input tokens for instructions, questions, chunks, and history before choosing a model.
  • Add a 10-20% safety buffer to account for uneven document sizes and metadata.
  • Use the RAG Token Budget Calculator to validate architecture choices against context limits.
  • Truncation in production is expensive; fixing token budgets in design is cheap.

The Bottom Line

Skipping the math on token budgets isn't just risky, it's a guaranteed way to ship a broken product. Plan your context limits first, or prepare for silent failures later.