Deterministic fixtures are necessary but insufficient for validating AI workflows. They prove the mechanism works, but they rarely catch the messy, real-world edge cases that trip up new versions. A recent incident highlighted this gap: a prompt change passed every fixture test but would have double-emailed twelve customers in production. The solution? The shadow run.

The Fork: Duplicating Events Without Duplicating Chaos

The shadow run architecture is deceptively simple. Every production workflow starts with a trigger, usually a Webhook or Schedule. In the shadow setup, the first step after the trigger is a fork that duplicates the event. Branch A runs the current production version (v1) and executes real side effects like sending emails or updating trackers. Branch B runs the new version (v2) but with every effectful node replaced by a stub.

Stubbing Side Effects: Intent Over Action

The critical trick is replacing side effects with log rows. Instead of firing a Gmail send or a Slack post, the shadow branch writes a record describing the intended action. This includes the event ID, workflow version, node name, effect type, target, and a payload hash. The shadow stub must be incapable of firing the effect, not just configured not to. A tag linter ensures zero nodes in the shadow branch are tagged as effectful, preventing accidental production touches.

The Divergence Query: Catching Bugs Before They Ship

After processing 50 live events, a SQL query compares the 'live_log' (what v1 did) against the 'shadow_log' (what v2 would have done). The query identifies three classes of divergence: missing effects, extra effects, and changed payloads. 'Missing' effects are always bugs. 'Extra' effects are the highest risk and require manual review of the actual payload. 'Changed' payloads are usually improvements but are still sampled to ensure they aren't regressions.

Cost and Control: Bounding the Experiment

Shadow runs double LLM spend because every event passes through two model stacks. To control costs, the fork node counts events and stops duplicating after 50. This cap, enforced mechanically by the workflow deactivating itself, limits extra model costs to roughly $2 per version. This is a negligible price compared to the cost of refunds and apology time from a production bug like the double-email incident.

Key Takeaways

  • Fixtures test mechanics; shadow runs test behavior against live traffic.
  • Replace side effects with stubs that log intent, ensuring no real-world impact.
  • Use a divergence query to identify missing, extra, or changed effects.
  • Bound the run to 50 events to control LLM costs while catching edge cases.

The Bottom Line

Don't let fixtures give you a false sense of security. If you're shipping AI workflows, you need to see what they would have done to real users before you let them do it. The shadow run is the safety net that turns 'it passed tests' into 'it's ready for production.'