If you've ever shipped an AI feature to production only to find it behaving differently a week later with no explanation, you're not alone. A new tutorial making the rounds on DEV.to argues that the real culprit isn't always the modelβ€”it's everything around it that's never been versioned.

The Versioning Blind Spot

The core problem is straightforward: most AI teams track the model name (GPT-4o, Claude 3.5 Sonnet, etc.) and call it done. But the tutorial makes a compelling case that prompts, JSON schemas, fallback rules, tool definitions, and retry limits are all deployment artifacts in their own right. Change any of them, and you've shipped codeβ€”probably without a commit message.

What Actually Needs Versioning

The author breaks down what belongs in your version history alongside model changes: the exact prompt text (not just "the customer support prompt"), output JSON schemas and their versions, fallback chain configurations when one model fails, tool definitions for agentic workflows, retry limits and backoff strategies, and temperature/top-p sampling parameters. Each of these can silently alter production behavior without touching a single line of model-related code.

Treating Prompts as First-Class Deployments

The practical takeaway is to adopt the same rigor you'd use for application code: commit prompt changes with meaningful messages, tag releases that include both model versions and prompt configs, maintain diffs so you can trace exactly why outputs changed last Tuesday, and treat your entire inference pipeline configuration as a deployable artifact. Some teams are already doing this by storing prompts in YAML or JSON alongside their infrastructure-as-code.

Key Takeaways

  • A prompt change IS a deploymentβ€”treat it like one with proper versioning discipline
  • Track more than model names: schemas, fallback rules, retry configs, and tool definitions all matter
  • Maintain diffs between prompt versions to diagnose mysterious production regressions
  • Store your entire inference configuration as code, not just notes in Slack

The Bottom Line

The gap between "works on my machine" and "works in production" is wider than most teams realize because they've been versioning the wrong things. If you want predictable AI behavior, start treating prompts like the deployments they already are.