The standard approach to multi-index RAG agents is broken. Developers Sankeerthan Kasilingam recently shared a post-mortem on Medium revealing that a typical orchestrator agent, relying on an LLM to select search tools, failed to route the majority of user queries to the correct index. By ditching the agent-based routing for a deterministic 'query planner' inspired by database engines, the team skyrocketed routing accuracy to 98% while reducing latency. This isn't just a bug fix; it's a fundamental shift in how we should architect agentic retrieval systems.
The Failure of Agent-Based Routing
The original system used a standard orchestrator pattern: a front-end agent with one tool per index (Handbook, Weekly Reports, Store Directory). The LLM read tool descriptions and decided which to call. Kasilingam's team tested this against a 'golden dataset' of 366 conversation turns. The result? The orchestrator only sent 62% of questions to the right place. Adding a simple keyword classifier didn't help much, hitting only 61%. The core issue was that the model was conflating two distinct tasks: understanding the user's intent (routing) and executing the search (retrieval). When a user asked, 'Can I install this tool before the next release?', the word 'release' triggered a false positive for the Weekly Reports index, even though the question was about the Handbook.
Borrowing From PostgreSQL
The solution came from looking at how databases handle complex queries. PostgreSQL doesn't execute SQL directly; it uses a query planner to generate an execution plan first. Kasilingam applied this logic to the RAG agent. Instead of an agent deciding which tool to call, a single LLM call now generates a structured JSON plan. This plan breaks down complex messages into atomic questions, rewrites follow-ups (e.g., changing 'who leads it?' to 'who leads Project Falcon?'), and assigns each sub-question to a specific domain. Crucially, the execution of this plan is handled by plain, deterministic code, not an LLM loop. This separation of concerns allowed the team to test the routing logic independently of the retrieval logic.
Solving the Temporal Problem
Routing accuracy was just the first hurdle. Even with 98% correct routing, answer accuracy for Weekly Reports remained low because the search engine couldn't distinguish between 50 different versions of the same report. The new planner doesn't just output a domain; it outputs time-sensitive parameters like 'as_of', 'from', and 'to' dates. The ingestion pipeline was also updated to attach trusted metadata to every document, replacing unreliable file names with verified dates. This ensured that when a user asked about 'current' status, the agent pulled the newest report, not the most semantically similar one from March. This shift corrected answer accuracy from 23% to 77%.
Key Takeaways
- Separate Routing from Retrieval: Do not let an LLM decide which tool to call in a loop. Have it generate a plan, then use code to execute it.
- Route Questions, Not Messages: Complex user inputs often contain multiple intents. Break them down into atomic questions before routing.
- Metadata is King: For temporal data, semantic search is insufficient. You must embed structured time and version metadata into your documents and plans.
- Deterministic Fallbacks: Use the LLM for understanding, but keep the execution path simple and testable. If the planner fails, fall back to keyword rules, not another agent.
The Bottom Line
We are over-engineering agentic routing. Kasilingamβs approach proves that a single, well-structured LLM call to generate a JSON plan, followed by deterministic code execution, outperforms complex agent loops. Stop letting your LLM guess which tool to use; make it tell you what to do, and let the code do the rest.