In a recent post on DEV.to, developer phpboyscout detailed an experiment using three distinct LLM providers—Anthropic, Gemini, and OpenAI—to extract structured data from a long-running Dungeons & Dragons campaign. The goal was to build 'Scout,' a tool that turns session transcripts into a canonical database of invented proper nouns and promises. While the multi-model approach seemed like overkill for a hobby project, the author discovered that the complexity wasn't in the voting mechanism, but in the fundamental ambiguity of what constitutes a single fact.

The Problem with Invented Entities

The core challenge Scout faces is that tabletop campaigns rely entirely on invented entities. Unlike news extraction, where models can lean on world knowledge (e.g., knowing Paris is a city), D&D transcripts contain names like 'candlefoot' or 'kettlesteam' that have zero prior existence. A model cannot rely on pre-training to disambiguate whether these are people, places, or items. Extraction is a 'write' operation to a permanent store; a single error here poisons all downstream queries, creating a scenario where confidently wrong data looks identical to correct data six months later.

Why Three Models Didn't Solve It

The developer initially assumed that majority voting across three providers would resolve extraction errors. However, the models did not fail independently in a way that made voting straightforward. Gemini, for instance, exhibited a tendency to hallucinate from source material rather than sticking to the transcript, a behavior noted in similar tools like NotebookLM. More critically, the models agreed on the events but disagreed on the 'granularity' of the facts. One model might bundle a location with a cost, while another separates them, leading to votes that are technically conflicting but substantively overlapping.

The Cost of Redundancy

Running three inference calls per session significantly impacts unit economics. The author noted that early prototypes used a local Claude backend to leverage an existing subscription, highlighting that 'a subscription you've already paid for is the best price there is.' However, shipping a multi-model panel without understanding the per-session cost is risky. The experiment revealed that the voting logic itself was trivial—about forty lines of code—while the reconciliation of differing fact definitions was the true engineering hurdle.

The Roster Is Disposable

The specific choice of Anthropic, Gemini, and OpenAI is treated as an implementation detail rather than a core design principle. The developer argues that the architecture must be model-agnostic because providers deprecate models and change pricing on short notice. By abstracting the providers behind a unified interface (go/chat), the system allows for swapping members without rewriting the extraction logic. This dogfooding approach exposed rate-limiting gaps in the developer's own modules, which were then filed upstream.

Key Takeaways

  • Fact granularity is the primary bottleneck in LLM-based data extraction, not model accuracy.
  • Multi-model voting requires sophisticated reconciliation logic to handle non-identical but overlapping fact bundles.
  • Prototyping with existing subscriptions (like Claude) offers a cost-effective way to test multi-model pipelines before committing to API costs.
  • Architectural decoupling of LLM providers is essential to survive rapid model deprecations and pricing changes.

The Bottom Line

This project is a sobering reminder that LLMs are not magic truth machines; they are probabilistic parsers that struggle with semantic ambiguity. The real work in AI engineering isn't picking the 'best' model, but defining the schema so clearly that different models can agree on what a 'fact' actually is.