AI coding agents are failing a critical validation skill: writing tests that actually check requirements instead of just mirroring existing code. A new benchmark called SpecGuardBench, detailed in arXiv:2610.16870, shows that even the top-performing agents achieve only 63% faithfulness when generating validation harnesses. The core issue is that agents tend to copy the behavior of the reference implementation they are shown, rather than strictly adhering to the natural language specification. When the reference code contains bugs, the agents propagate those bugs into their test suites, effectively approving incorrect behavior.
The Mirror Trap: Faithfulness Drops When Reference Code Is Buggy
The researchers introduced a specific trap to test this behavior: showing agents a buggy implementation as the reference code. If agents were truly working from the requirement, the quality of the reference code should be irrelevant. However, the data shows a massive dependency on the code itself. For instance, OpenCode using GLM-5.3 saw its faithfulness score plummet from 68% with a correct reference to just 39% with a buggy one. Similarly, Claude Code using Claude Opus 5 dropped from 52% to 21%. This 19 to 31 point drop proves that current agents trust the visible code more than the written spec.
Benchmark Results: Codex and OpenCode Lead, But Still Fail
The benchmark tested 291 Python tasks across eight types of verification logic, including functional postconditions and interaction protocols. OpenCode with GLM-5.3 led the pack with 62.9% faithfulness, catching 74.2% of bugs but also rejecting 15.1% of valid code alternatives. Codex with GPT-5.6-sol followed closely at 61.9%, catching more bugs (81.8%) but rejecting significantly more valid code (23.0%). This trade-off highlights a fundamental flaw: agents are not distinguishing between what the code does and what the spec requires. A faithful harness must reject the bug and accept every correct version, a standard no agent currently meets consistently.
SpecGuard: A Browser-Based Tool for Private Model Testing
To make this issue accessible, developer vishalmysore released SpecGuard, a free, in-browser recreation of the benchmark. It allows users to test local models via WebGPU without sending data to external servers. The tool includes 16 JavaScript tasks and features a 'mirror trap' mode to instantly visualize how much a model relies on reference code. It also provides baselines, such as a 'hand-written oracle' that achieves 100% faithfulness, proving that the tasks are solvable, and a 'mirror the reference' baseline that scores 0% because it rejects all valid alternatives. This tool integrates with ZeeSpec, an open-source spec builder, to help developers verify if their agents can turn vague requirements into precise checks.
Key Takeaways
- AI agents prioritize reference code over natural language specs, leading to false positives for bugs.
- Faithfulness scores drop by up to 31 points when agents are shown buggy implementations.
- SpecGuard offers a private, browser-based way to test local models for this specific failure mode.
- Existing tests often pass buggy code, so agents mimicking those tests inherit the errors.
The Bottom Line
We are automating the very skill that makes automation dangerous. Until agents can separate the spec from the implementation, their 'tests' are just rubber stamps for whatever code was already there, and we need tools like SpecGuard to expose this blind spot before it scales.