If you're building AI agents right now, chances are you've shipped at least one skill without any real testing framework behind it. Philipp Schmid from Google DeepMind wants to change that. In his talk titled "Don't Ship Skills Without Evals," Schmid cuts through the hype and delivers a blunt message: everyone is writing skills for AI agents, almost nobody is testing them properly, and that's a problem waiting to bite you in production.

The Non-Determinism Problem

The core issue Schmid identifies isn't that developers are lazy—it's that LLMs behave differently every time. Unlike traditional software where if (x) { y } always does the same thing, an LLM might give you subtly different outputs on each run. Eyeballing the output during development and thinking "looks right" doesn't cut it when you're deploying to real users with real stakes.

What Evaluation Frameworks Actually Look Like

The talk dives into what proper skill evaluation actually means in practice. Rather than treating AI agent behavior as unmeasurable, Schmid advocates for building test suites specifically designed for non-deterministic outputs—using techniques like sampling multiple runs, defining behavioral assertions rather than exact string matches, and creating golden datasets that capture the range of acceptable responses.

Building a Culture of Measurable Skills

Schmid's argument extends beyond tooling into developer culture. The skills ecosystem has exploded with everyone publishing prompts and agent configurations, but without evals, there's no way to know if that "amazing" skill you installed last week actually works as advertised—or if it was just lucky output on a demo.

Key Takeaways

  • Treat AI skills like critical infrastructure: ship them with test coverage or don't ship them at all
  • Define behavioral tests, not exact output matching—accept the range of correct responses
  • Use sampling and statistical measures to catch regressions across LLM versions
  • The community needs shared eval benchmarks to separate hype from actually-working solutions

The Bottom Line

The skills economy is booming, but without evaluation infrastructure, we're essentially flying blind. Schmid's talk should be required watching for anyone building or consuming AI agent extensions—the industry can't mature if we don't start measuring whether our tools actually work.