The debate over lexical versus semantic retrieval for tool calling just got a definitive data point. In a follow-up experiment published on DEV.to, developer Steefjan Wiggers swapped BM25 for text-embedding-3-small on Azure OpenAI to re-evaluate tool recall. The results confirm that while embeddings dramatically improve performance on paraphrased requests, a simple alias sentence in the tool description remains the most efficient solution for controlled environments.

Embeddings Fix the Paraphrase Catastrophe

The original experiment showed BM25 failing miserably on paraphrased requests, scoring just 5 percent recall at k=5. The rematch, using cosine similarity on embedded vectors, jumped that number to 76 percent for the same tasks. This validates the core theory of semantic search: while a user saying I want to cancel my booking shares almost no tokens with a tool described as deletes a reservation, the two sentences live close together in embedding space. The k-sweep also behaves differently; widening the shortlist from 3 to 10 tools now buys significant recall (59 to 94 percent), whereas with BM25, the correct tool was often absent from the ranking entirely.

Verbosity Helps Semantics, But Hurts Exactness

Surprisingly, verbose descriptions now yield a 9-point improvement in paraphrase recall over terse ones (82 percent vs. 73 percent at k=5). This contradicts previous findings where verbosity offered zero lexical benefit. However, embeddings introduce a new failure mode: they blur exact matches. When users echo the tool's own vocabulary, BM25 scored 97 to 100 percent, while embeddings dropped to 91 to 97 percent. This divergence provides concrete evidence for hybrid retrieval systems, which can leverage the strengths of both approaches.

The Alias Sentence Remains Unbeatable

Despite the gains from embeddings, the alias sentence interventionโ€”adding just 13 tokens of synonyms to each tool descriptionโ€”achieved 99 to 100 percent recall across all conditions. This result held true for both BM25 and embedding-based retrieval. For developers who control their tool descriptions, the operational cost of maintaining a vector index, embedding model, and lookup infrastructure is hard to justify when a static text addition solves the problem. The embedding pipeline is best reserved for gateways or catalogs fronting third-party MCP servers where you cannot edit the source descriptions.

Key Takeaways

  • Embeddings increase paraphrase recall from 5% to 76% compared to BM25.
  • A 13-token alias sentence achieves 99-100% recall with zero infrastructure overhead.
  • Embeddings degrade exact-match performance by 6-9 percentage points compared to lexical matching.
  • Hybrid retrieval is the optimal path for mixed query types.

The Bottom Line

Fix your words before you buy a vector database. If you own the tool descriptions, aliases are a free lunch; if you don't, embeddings are the only way to avoid a catastrophic failure rate on natural language queries.