Building a Retrieval-Augmented Generation (RAG) chatbot that survives real-world usage is less about model selection and more about the unglamorous mechanics of data retrieval. A new technical breakdown from DEV.to details a production-grade implementation using Amazon Bedrock, NestJS, and PostgreSQL that exposes a critical truth: most RAG failures are silent. The API returns 200 OK, dashboards stay green, and Claude generates text, but the answers gradually drift from reality as documents change and user queries grow complex.

The Silent Degradation of RAG Systems

The article, authored by developer hieu_le, highlights how quality decays without obvious errors. When a user asks about error code E-4021, the system might confidently return documentation for E-4012 because the retrieval step failed to prioritize the correct chunk. Recently updated documents may vanish from search results, and prompt caching might be enabled without actually reducing costs. The core issue identified is that teams often obsess over prompt engineering while ignoring the underlying search architecture. In enterprise applications, success or failure hinges on retrieval quality, not the LLM's raw intelligence.

Hybrid Search with Reciprocal Rank Fusion

To combat retrieval weakness, the implementation relies on a hybrid search strategy using Reciprocal Rank Fusion (RRF). The system runs two parallel searches: a dense vector search using HNSW indexes in pgvector and a keyword search using PostgreSQL’s GIN indexes. These results are merged using RRF with a k-value of 60, ensuring that chunks appearing in both semantic and keyword matches rise to the top. For example, a query like 'warranty 12 months' benefits from the keyword branch catching exact terms, while the dense branch handles paraphrased questions like 'How long is the product warranty?'

Chunking and Embedding Consistency

The ingestion pipeline is equally rigorous, employing a two-level chunking strategy that splits documents by headings first, then by length (1200 characters with 120-character overlap). Crucially, the system injects headings into the chunk content before embedding. This ensures that a chunk about 'Warranty period' carries that semantic weight in its vector representation. The stack uses Cohere Embed Multilingual v3 via Amazon Bedrock, generating 1024-dimensional vectors. A common pitfall noted is the mismatch of input_type parameters; documents must use search_document and queries must use search_query to ensure they live in the same vector space.

Key Takeaways

  • Retrieval quality, not prompt engineering, determines RAG success in production.
  • Hybrid search combining HNSW vectors and GIN keyword indexes via RRF significantly improves accuracy for mixed query types.
  • Header injection during chunking prevents 'averaged out' embeddings and improves relevance for section-specific questions.
  • Silent failures, such as stale chunks or mismatched embedding types, degrade quality without triggering infrastructure alerts.
  • Using a single database (PostgreSQL + pgvector) for both vector and full-text search eliminates synchronization complexity.

The Bottom Line

Stop treating RAG as a simple 'plug-and-play' LLM wrapper. If your retrieval architecture isn't explicitly designed with hybrid search and strict embedding consistency, your 'intelligent' chatbot is just confidently guessing.