If you've shipped any LLM-powered feature in the past year, you've probably been burned by this scenario: your offline eval scores look great, you ship the change, and then user satisfaction tanks. The disconnect between synthetic benchmarks and real-world performance has become one of the most frustrating problems in AI product development—and a new guide from Rishi Kora on DEV.to makes a compelling case that online A/B testing is the answer.

Why Offline Evals Keep Lying to You

Traditional evaluation pipelines for LLMs rely on curated datasets, human annotations, and synthetic benchmarks. These approaches have their place, but they share a fundamental flaw: they're measuring proxy behavior rather than actual user outcomes. Kora argues that no matter how sophisticated your offline eval framework becomes, it can never fully capture the diversity of real prompts, edge cases, and usage patterns your users will encounter in production. The guide walks through common failure modes where high benchmark scores don't translate to improved user experience—particularly around tone, consistency, and task completion rates.

Setting Up Your First Online Experiment

The practical heart of Kora's guide focuses on designing experiments that actually answer the question: 'Did this LLM change make things better for real people?' The approach centers on defining an Overall Evaluation metric upfront—what the author calls a 'true north' for measuring user impact. This requires thinking carefully about what success looks like beyond simple engagement metrics: task completion, error rates, support ticket volume, and explicit feedback signals all play a role. The guide provides concrete recommendations for experiment duration, sample sizing, and statistical rigor needed to detect meaningful differences in LLM behavior at scale.

Avoiding Common Pitfalls

One of the more valuable sections addresses pitfalls that trip up even experienced teams: treating A/B tests as binary win/lose decisions rather than learning opportunities, failing to account for novelty effects when users encounter changed AI behavior, and not stratifying results across different user segments. Kora emphasizes that LLM features often affect different cohorts dramatically differently—what works beautifully for power users might frustrate newcomers, and vice versa.

The Infrastructure Challenge

Running statistically valid experiments on LLM features isn't trivial from an infrastructure perspective. You need routing logic to direct traffic, logging systems that capture both the input context and model outputs, and analytics pipelines that can correlate AI behavior with downstream user outcomes. The guide touches on practical approaches for teams using feature flags and gradual rollouts to manage risk while gathering real signal about model changes.

Key Takeaways

  • Offline evals measure proxy behavior, not actual user outcomes—no benchmark suite replaces real traffic
  • Define an Overall Evaluation metric before shipping any experiment; don't let vanity metrics drive decisions
  • Account for segment differences in your analysis—LLM features affect user cohorts unevenly
  • Statistical rigor matters: underpowered experiments waste engineering time and can mislead product decisions

The Bottom Line

The gap between what benchmarks measure and what users actually experience is the dirty secret of AI product development right now. Kora's guide isn't revolutionary—it's practical operational wisdom that too many teams are still ignoring because setting up proper online experiments feels harder than trusting their eval suite. It's not. And if you're shipping LLM features without this kind of validation loop, you're essentially flying blind and hoping for the best.