Building a Retrieval-Augmented Generation (RAG) system is often treated as a solved problem in the dev community, but a recent deep dive by developer Jerry Qian highlights how fragile the underlying evaluation metrics can be. Using a standard stack of Bun, TypeScript, SQLite with sqlite-vec, Voyage embeddings, and Claude for generation, Qian constructed a ~700-line CLI tool that appeared functional but was fundamentally flawed in how it measured success.

The Illusion of Competence

The core issue wasn't the code itself, which Qian describes as "a weekend's work" with nothing particularly interesting in the stack. The failure lay in the evaluation phase. The system returned answers with citations, creating a veneer of accuracy that masked deeper retrieval failures. This is a common trap for builders: if the LLM generates a plausible-sounding answer with a citation, we assume the retrieval pipeline worked correctly, even when the context was irrelevant or incomplete.

Why Standard Metrics Fail

Qian's experience underscores a critical gap in current RAG development practices. Most developers rely on high-level metrics like answer correctness or faithfulness, which are often evaluated by LLMs themselves. This creates a circular logic where the generator validates the retrieval, ignoring whether the actual chunks retrieved were the most relevant. Without granular inspection of the retrieval stepβ€”specifically looking at vector distance and chunk rankingβ€”the evaluation is essentially lying to you about the system's true performance.

The Builder's Takeaway

For teams deploying RAG in production, this serves as a warning against trusting end-to-end evaluation scores without auditing the retrieval layer. You need to inspect the raw vector search results, not just the final generated text. If your evaluation doesn't explicitly measure whether the correct chunks were retrieved before generation, your metrics are likely inflated. The "twice" in the title suggests that even after fixing the first layer of evaluation errors, a second, more subtle failure mode emerged, likely related to how context is synthesized or ranked.

Key Takeaways

  • Standard RAG evaluation metrics can be misleading if they don't separately audit the retrieval and generation phases.
  • A stack of Bun, TypeScript, sqlite-vec, and Voyage can produce plausible but inaccurate results if evaluation is superficial.
  • Developers must inspect vector search rankings and chunk relevance directly, rather than relying solely on LLM-generated citations.
  • Plausible-sounding answers with citations are not proof of correct retrieval; they often mask underlying pipeline failures.

The Bottom Line

If you aren't explicitly auditing your vector search rankings, your RAG metrics are fiction. Stop trusting the LLM's confidence score and start looking at the raw retrieval data.