Stop tuning your RAG query rewrites by vibes. A new experiment published on DEV.to by developer dev48v demonstrates that recall is a fundamentally flawed metric for evaluating synonym expansion in BM25 indices. The core issue is mathematical: since expansion only adds terms to a query, it creates a superset of retrieval possibilities, meaning recall can never decrease. If you are using recall to decide whether a rewrite strategy is working, you are approving bad code by default.

The Precision Myth and Vocabulary Mismatch

The author initially hypothesized that synonym expansion would raise recall but lower precision, a common trade-off in information retrieval. The data proved otherwise. When applied to a real corpus with hand-labelled relevance, the 'expand' strategy increased recall from 0.688 to 0.938 and precision from 0.375 to 0.542 simultaneously. This occurred because the added synonyms (like 'automobile' for 'car') were not noise, but the exact missing vocabulary needed to bridge the gap between user queries and document content. The trade-off everyone quotes is a property of synonym quality, not expansion itself.

HyDE Fails Due to Filler, Not Method

HyDE (Hypothetical Document Embeddings) performed poorly, dropping MRR from 0.938 to 0.323. The experiment reveals the mechanism is mundane: pseudo-answers are stuffed with generic filler terms like 'this', 'document', and 'process'. In this specific corpus, two documents were majority filler, causing HyDE queries to retrieve irrelevant boilerplate. The control test is critical: when filler terms were stripped from the pseudo-answer, HyDE’s MRR matched the raw query exactly. This proves the failure was due to corpus-specific boilerplate, not the HyDE methodology itself, highlighting the necessity of measuring against your own data.

Why Recall Is a Tautology

The code for expansion shows why recall is a trap: expand pushes synonyms into an output array without removing existing tokens. A superset of query terms can only retrieve a superset of documents. Therefore, the statement 'our rewrite improved recall' is nearly tautological for any additive rewrite. In the experiment, even 'wrong synonyms' maintained a recall of 0.688, identical to the baseline, while damaging MRR. Precision@3 remained stable at 0.375 because wrong synonyms were rare terms that didn't pull in enough documents to hurt the top-3 cutoff, but the damage appeared in the ranking of the first relevant hit.

Key Takeaways

  • Recall is a poor metric for additive query rewrites; it can only go up or stay flat.
  • Synonym expansion can improve both recall and precision if it addresses genuine vocabulary mismatches.
  • HyDE fails in corpora with high boilerplate due to filler terms, not the algorithmic approach.
  • MRR is a superior metric for RAG pipelines because generators only see truncated, ranked lists.
  • Always verify BM25 implementations against dense matrix calculations to catch indexing errors.

The Bottom Line

If you are building RAG infrastructure, stop celebrating recall gains. They are mathematically inevitable for expansion strategies and often meaningless. Focus on MRR and precision@k, and always test against your specific corpus’s noise profile.