Developers relying on regeneration as a sanity check for AI outputs are missing the most dangerous failure mode: lone-model hallucinations. A new divergence study conducted by MultiO reveals that asking the same model twice only verifies internal consistency, failing to catch errors that are unique to that specific architecture. While re-asking confirms a model isn't randomly fluctuating, it completely ignores the fact that the model might be confidently wrong in a way its peers are not.

The Illusion of Consistency

The study sent 24 identical questions to five major LLMs: GPT, Claude, Gemini, Grok, and Llama. Across the 24, there were 52 material divergences, and exactly half of them (26) involved a lone model: 14 claims exactly one model made, and 12 stances exactly one model took against the rest. To test the efficacy of re-asking, researchers ran two of these divergent questions three times each through a single model, ensuring cache-ineligible fresh samples. The result was stark: the model never changed its verdict. The variance was purely in wording, not substance, proving that a lone error is deterministic within that model's current context.

Why Cross-Model Validation Matters

The structural problem is that disagreement in this dataset lives between models, not within a single model's sampling distribution. When a model makes a lone claim, re-asking it simply replays that same error with the same high confidence. For example, in a debate about monorepos, one model listed simplified CI/CD as a monorepo benefit while every other answer treated CI as the main cost. Its own answer elsewhere said pipelines get more complex, and it also recommended Git submodules and Bazel, a stance no other model took. A second sample from that same model would have repeated the contradiction, not resolved it. In contrast, comparing it against other models would have immediately flagged the outlier.

The Value of Outliers

This isn't just about catching errors; it's about preserving valuable insights. The study found that of the 14 sole-source claims, 8 were the highest-value line in their set and 3 were wrong or invented. Synthesizing answers by averaging the panel or deleting outliers risks losing these critical, falsifiable conditions. A lone voice might be the only one stating a necessary caveatβ€”such as the condition that walk-in-only can still win under roughly 20 seatsβ€”that the other models omitted entirely. Without cross-model comparison, developers are forced to choose between trusting a potentially flawed consensus or discarding unique, high-value data.

Key Takeaways

  • Re-asking the same model only checks for sampling variance, not factual accuracy against other architectures.
  • 52 material divergences were identified in the study, with 26 involving a lone model.
  • Cross-model validation is the only method that adds information; single-model re-asks merely confirm internal consistency.
  • Outlier claims are often high-value (8 of 14 sole-source claims were the best line in their set), making synthesis-by-averaging a risky strategy.

The Bottom Line

Stop treating re-generation as a quality assurance tool for accuracy; it only confirms that a model is consistently wrong. For any decision with financial stakes, the only reliable validation is cross-model comparison that exposes where architectures diverge, not where they repeat themselves.