The debate over how much context AI agents need has just gotten a lot more interesting. A new benchmark published by developer jackymencz tests three distinct strategies for feeding software code to GPT-6 Astra: RAW (full source code), IR (Intermediate Representation/summary), and a hybrid IR+RAW approach. The results suggest that the hybrid model, which starts with a structured map and escalates to raw code when necessary, achieved perfect accuracy on 87 questions while using significantly fewer input tokens than the brute-force full-code method.

The Three Investigators

The benchmark treats the AI agent like an investigator examining a software project. The first approach, RAW, dumps the entire codebase into the context window, akin to giving a detective every page of a case file regardless of relevance. The second, IR, provides a structured summary of facts—such as exposed functions, HTTP routes, and environment variables—similar to handing over a building floor plan instead of walking every room. The third, IR+RAW, combines these: the agent starts with the structured map but retains the ability to query the original source code if the summary lacks sufficient detail.

Benchmark Results and Token Efficiency

Across 87 questions, the RAW approach answered 84 correctly with zero unresolved issues, consuming an estimated 279,476 input tokens. The standalone IR approach answered 82 correctly but left five unresolved, using only 58,549 tokens. The hybrid IR+RAW approach achieved the highest accuracy, answering all 87 questions correctly, while using 80,340 estimated input tokens. This represents a 71% reduction in estimated input tokens compared to the RAW approach, demonstrating that targeted retrieval of raw evidence can maintain accuracy while drastically cutting context overhead.

The Compression Paradox

One counter-intuitive finding from the study is that structured summaries are not always smaller than the original code. For small files, such as a 27-line HTTP server, the IR representation actually required more tokens (272) than the raw source (163) due to the overhead of labels and field names. However, as file size increased, the IR approach became exponentially more efficient. For a 951-line file, the raw source required 7,186 tokens, while the IR representation dropped to just 498. This highlights that the hybrid approach’s efficiency scales with codebase size, making it particularly viable for large projects where full-context windows are prohibitive.

Key Takeaways

  • The hybrid IR+RAW approach achieved 100% accuracy on the benchmark, outperforming both full-code (96.6%) and summary-only (94.3%) methods.
  • Hybrid agents used 71% fewer estimated input tokens than full-code agents, offering significant cost and latency advantages for large codebases.
  • Structured summaries (IR) can be larger than raw code for small files, but offer massive token savings for larger, complex projects.
  • The benchmark relied on estimated token counts (length/4), not exact provider billing, so real-world cost savings may vary.

The Bottom Line

This benchmark validates a critical engineering principle: don't force AI agents to choose between ignorance and everything. The hybrid model’s ability to escalate from summary to source code when evidence is insufficient is the smartest path forward for scalable, accurate AI agents. It’s not about making the model smarter; it’s about giving it better investigative tools.