Stop over-engineering your AI agent’s retrieval system. A new benchmark from Tobias Uhlig of Neo.mjs reveals that simple grep searches, when ranked correctly, can outperform sophisticated Retrieval-Augmented Generation (RAG) indexes. The test, conducted on November 16, 2026, replayed 40 real questions from an AI maintainer’s query log against two repositories containing 7,690 files. The results were stark: ranking grep hits by distinct question words scored 30 out of 40, while BM25 ranking hit 34 out of 40. In contrast, the team’s existing RAG index scored only 26 before fixes, and plain vector search lagged at 15.

The Power of Proper Ranking

The core finding is that how you rank results matters more than the tool you use to find them. The team tested three ranking methods for ripgrep (grep) results. The naive approach—sorting by total match count—performed poorly, scoring just 10/40 because long files with repeated common words dominated the results. However, ranking files by how many distinct question words they contained jumped the score to 30/40. This method prioritizes files that define the concepts in the query rather than just mentioning them repeatedly. Using textbook BM25 parameters, which weigh rare words higher and penalize long documents, the grep baseline achieved a 34/40 score, tying the repaired RAG index.

Why the RAG Index Failed

The team’s RAG index, which uses vector embeddings from qwen3-embedding-8b and Chroma, initially underperformed. Plain vector search scored 15/40 because 115 of its top 200 results were historical tickets and pull request conversations rather than current code. The index prioritized history over the present state of the codebase. After debugging, the team identified three specific flaws: a guide boosting logic that awarded points for question words even if the guide didn’t contain them, a base class collecting boosts from every subclass match, and a file lookup that failed to reach installed package sources. Fixing these issues raised the RAG index’s score to 33/40, but it still trailed the simple BM25 grep baseline by one point.

The Staleness Trap

A critical vulnerability of RAG indexes is data staleness. The Neo.mjs index was three months out of date on source files, meaning it missed answers in newer code. For the six questions where the answering file was newer than the index, the repaired RAG index found only 1 correct answer, while BM25 over the checkout found all 6. This highlights a fundamental operational risk: an index is only as good as its last ingest. A search over the current checkout is always up-to-date, whereas an index requires rigorous maintenance to remain relevant. The test showed that combining both methods—using BM25 for current code and RAG for historical context—achieved a perfect 40/40 score.

Key Takeaways

  • Ranking Strategy is King: Switching from count-based ranking to distinct-word or BM25 ranking improved grep performance from 10/40 to 34/40.
  • RAG Needs Maintenance: Vector indexes can score lower than lexical search if they are stale or if their ranking logic favors historical noise over current code.
  • Hybrid Approach Wins: Using both BM25 over the checkout and a well-maintained RAG index covered all 40 test questions, with neither tool alone achieving perfection.
  • Verify the Source: AI agents should be instructed to open the actual file before asserting how code works, as search results are leads, not proof.

The Bottom Line

Before you build a complex vector database for your coding agent, spend an evening writing a script to rank grep results with BM25. You might find that simple lexical search, properly tuned, is faster, fresher, and more accurate than your expensive RAG pipeline.