General-purpose LLMs are drowning in the specifics of niche industries, according to a new benchmark from Metal America that tested six leading models against 387 real-world construction questions. The study highlights a critical failure mode for enterprise AI: while models can articulate abstract concepts like snow load physics, they collapse when asked for company-specific operational truths, such as manufacturer delivery timelines or local permit requirements. The data confirms that without retrieval-augmented generation (RAG), even the most advanced models are effectively hallucinating business logic.

The Benchmark Methodology

The evaluation framework was rigorous, covering 13 distinct categories ranging from structural specifications and financing rules to installation processes. Six models were pitted against each other: Claude Opus 5.5, DeepSeek V4 Flash, Llama 4 Maverick, GPT-6.1 Sol, Mistral Large 3, and Gemini 2.5 Flash. Each question was tested under two conditions: standalone inference and inference with relevant source pages retrieved into the prompt. To ensure objectivity, written answers were graded by a separate model blind to the source, while numeric answers were scored against defined tolerances.

Retrieval Bridges the Gap

The results were stark. On average, models achieved only 19% accuracy when operating without retrieval. However, when provided with the correct context, accuracy surged to approximately 90%. Claude Opus 5.5 led the pack with 96% accuracy with retrieval, followed closely by DeepSeek V4 Flash at 94% and Llama 4 Maverick at 93%. Conversely, standalone performance was dismal across the board, with Mistral Large 3 scoring just 11% and Gemini 2.5 Flash managing 21%. The data suggests that model choice matters significantly less than data access; with retrieval, even smaller open-weight models outperformed the strongest closed models in standalone mode.

The Hallucination of Consensus

Perhaps the most dangerous finding was the phenomenon of 'consensus-wrong' answers. In several instances, all six models agreed on an answer that was factually incorrect for the specific company. For example, when asked if change orders could be added after a customer signed a concrete quote, every model answered 'yes,' reflecting general industry norms rather than Metal America’s specific policy. None of the models flagged their uncertainty or indicated they were guessing based on typical behavior rather than company rules. This silent failure mode poses a severe risk for automated customer-facing agents.

Corpus Quality Dictates Ceiling

The benchmark also revealed that retrieval systems are only as good as their underlying documents. The few errors that persisted with retrieval were traced back to contradictions within the company’s internal documentation, such as conflicting payment terms or fee structures. In these cases, the models faithfully reproduced whichever page they were shown, proving that RAG systems will amplify, not correct, messy data. The study argues that auditing the corpus for contradictions is a prerequisite to blaming model performance.

Key Takeaways

  • Evaluate systems both with and without retrieval to measure dependency on the corpus.
  • Use blind grading models to avoid bias in assessing written responses.
  • Look for consensus-wrong answers to identify where business rules deviate from industry defaults.
  • Audit document consistency before optimizing model parameters, as contradictory sources degrade RAG accuracy.

The Bottom Line

For enterprise AI, the model is the least important variable; the data corpus is the bottleneck. Stop tuning parameters and start fixing your documentation.