The most useful number published in AI this month did not come from a model launch, but from a data company. Garrett Lord, founder of Handshake AI, revealed that his training-data business reportedly reached roughly $1 billion in gross annualized revenue in about a year. This surge isn't driven by scraping more web text, but by a fundamental shift in what frontier labs are willing to pay for: expert judgment encoded in simulated work environments.

The Shift From Labeling to Authoring

Lord put the market for AI training data at roughly $10 billion today, up from about $1 billion three years ago, and forecast $100 billion within three years. However, the nature of the product has changed. Handshake no longer sells rows of labeled text; it builds simulated work environments—like a fake investment bank with email threads and data rooms—where domain experts author tasks and grading rubrics. Lord explicitly rejects the term "labeling," stating that "it's experts authoring data," a distinction that moves the labor market from students drawing boxes for $10 an hour to professionals building complex, high-fidelity work scenarios.

Why Small Businesses Are Sitting on Gold

Frontier models still struggle significantly with closed-domain expert work. On Handshake’s finance benchmark Atlas, where the task is building the kind of financial model a junior banker delivers, frontier models score below 20%. On Vials, its life-sciences benchmark, most models stay under 50% even when given tools. This gap exists because models lack the specific, nuanced judgment that defines "correct" in a professional context. Small businesses generate this exact asset daily: the quote that won the job, the email thread that de-escalated a customer, and the corrections made to AI drafts. Unlike the lab environments that take days to build, your version of this data already exists and evaporates if not captured.

Implementing the 20-Minute Answer-Key File

You don't need a team of 800 to build a proprietary eval set. Start by picking one recurring job family, such as quote replies or weekly reports, and collect five real examples of inputs and your actual finished outputs. De-identify sensitive data, then write one line per example explaining what makes the output correct—this is the judgment layer you cannot buy. Use these examples to grade AI drafts before they leave your hands and to guide prompts by pasting the closest example as a reference. This file remains valuable precisely because it is private; public test sets leak into training data, but your specific business logic stays unique to you.

The Correction Log as a Compounding Asset

The second half of this strategy is the correction log. Every time you fix an AI draft, spend five minutes writing down what was wrong and the rule it broke. For example, "Used wrong payment terms—always check against signed contract." After ten entries, this log becomes part of your standard prompt, pre-answering recurring failure patterns. This creates a compounding asset: examples for the model to imitate and corrections for it to avoid. While labs use complex rubrics with 146 criteria, your simple two-column log captures the same signal without the overhead.

Key Takeaways

  • The AI data market is shifting from raw volume to expert-authored, judgment-heavy environments, with Handshake AI hitting $1B ARR in a year.
  • Frontier models still fail at specialized tasks (sub-20% on finance benchmarks), creating a gap that proprietary business data can fill.
  • Small businesses can build effective eval sets by collecting five real examples and writing one-line justifications for why they are correct.
  • A daily correction log captures implicit rules, turning ad-hoc fixes into a compounding prompt asset that improves AI reliability over time.

The Bottom Line

The expensive input in AI right now is not compute or the newest model; it is real examples of finished work plus the expert judgment about what makes them right. Small businesses generate this asset every week but throw most of it away by failing to file it. Stop treating your corrections and best outputs as ephemeral; they are the proprietary data moat that frontier labs are paying billions to replicate.