Every AI coding team eventually hits the same wall: someone builds a fancy harness with pre-commit gates and instruction files, only to watch the agent pass the tests anyway. Or worse, the gate blocks the agent six times in a morning, leading to the inevitable question: are these guardrails actual engineering or just ritual? A new experiment published on Hacker News attempts to answer this with data rather than vibes, running 270 unattended coding agent sessions to see if scaffolding actually improves outcomes.

The Setup: Three Conditions, One Codebase

The experiment, conducted on October 13 and 14, 2026, used Claude Code 2.1.296 across three models (Sonnet, Opus, and Haiku) and seven tasks. The author tested three conditions: 'none' (raw agent), 'rules' (agent with an AGENTS.md instruction file), and 'gates' (rules plus a pre-commit gate, stop hook, and destructive command blocker). The key metric wasn't just whether the code passed hidden acceptance tests, but what the agent left behind when it declared the job 'done.'

The Result: Pass Rates Stayed Flat

Surprisingly, the guardrails did not improve the first-pass success rate. Across 90 runs per condition, the 'none' setup passed 90/90, 'rules' passed 89/90, and 'gates' passed 88/90. Statistical analysis found no significant difference. If you're adding complexity to your agent workflow expecting a higher immediate success rate, this data suggests you're wasting cycles. The model's baseline capability to pass basic tests was already near-perfect in this small-codebase context.

Where Guardrails Actually Paid Off

The real impact appeared in the 'defects at done'β€”issues that passed the test but broke project rules. On task T7, which required recurring invoices, 14 of 15 runs in the 'none' condition changed the database schema without adding the required migration test. With guardrails (both 'rules' and 'gates'), 0 of 30 runs made this mistake. Similarly, the large model (Opus) changed files outside the task scope in 10 of 35 'none' runs, but only in 2 of 35 'rules' runs and 1 of 35 'gates' runs.

The Cost and The Caveat

Guardrails didn't significantly impact cost. The total experiment cost $115.58, with Haiku averaging $0.06 per accepted change and Opus averaging $0.82. The harness added negligible cost. However, there's a major caveat: the 'read-only tests' rule in the instruction file caused agents to stop and ask for human intervention when they needed to modify existing tests. This accounted for all defects in the 'rules' and 'gates' conditions for task T4, effectively trading automation for safety.

Key Takeaways

  • Guardrails do not improve first-pass test success rates for capable models like Claude Sonnet/Opus in small tasks.
  • Guardrails drastically reduce 'hidden' defects like missing schema migration tests (14/15 errors to 0/30).
  • Strict instruction files can halt progress if they forbid necessary actions like updating existing tests.
  • Haiku is surprisingly cost-effective for fully specified small tasks, averaging $0.04-$0.06 per accepted change.

The Bottom Line

Stop measuring guardrails by pass rates. Measure them by the silence they keepβ€”the schema changes without tests and the scope creep they prevent. If you want clean code, you need the gate; if you want speed, you might not.

Source

This analysis is based on the blog post 'Do Guardrails Make AI Coding Agents Better? What 270 Runs Showed' published on secondtimeisabug.com.