In the underground of AI agent tooling, a new benchmark from developer Ces107 has flipped the script on how we evaluate code-review skills. After testing 18 seeded-bug pull requests across Claude Opus and Haiku, the results are clear: detection rates are a dead heat, but trust is not. The study, published on DEV.to, demonstrates that while modern models can find the same bugs, only specific skill architectures can prove them without human re-verification.

The Setup: High-Stakes Medical Integrations

Ces107, who builds correctness-critical software for hospital integrations (HL7, DICOM, FHIR) under strict EU medical-device rules, built this benchmark to answer a practical question: "Can I act on what the AI says without re-checking everything myself?" The test suite consisted of 14 PRs with production-grade bugsβ€”ranging from idempotency races that double-charge cards to silent bank amount zeroing via except Exception: return 0β€”and 4 clean PRs designed to tempt false positives. Crucially, 8 cases were generated by a separate agent to prevent overfitting, and every bug was confirmed at runtime.

Result 1: Detection Is a Dead Tie

Both the "prompt-only" reviewer (a standard marketplace checklist skill) and "Proven Code Review" (Ces107’s custom skill) identified 15 out of 15 seeded bugs across both Claude Opus and Claude Haiku. Both achieved 18/18 correct verdicts on clean PRs. This finding suggests that current frontier models are already capable enough to find common logic errors regardless of the wrapping prompt. If a vendor claims their skill "finds more bugs," they are likely marketing to a problem that no longer exists in detection alone.

Result 2: The Trust Gap is Massive

While detection was equal, the quality of the output diverged sharply. On Claude Haiku, the prompt-only reviewer backed only 14% of its findings with actual proof (a run test or command), compared to 76% for the Proven Code Review skill. The prompt-only approach also generated 25 "nits" (low-value comments) versus just 5 for the proven skill. On the faster, cheaper model, the gap in noise reduction and proof backing was nearly 5x. This is critical for CI pipelines where signal-to-noise ratio determines whether developers actually read the output.

Proof Over Prompts: How It Works

The "Proven Code Review" skill differs architecturally by mapping every caller of changed code, even those outside the PR diff, and requiring a failing test or command execution for every Critical and High finding. In one instance involving an open-redirect vulnerability, the skill didn't just flag the regex; it wrote and ran a test, pasting four specific URLs that landed users on evil.test after login, including a bypass (https://app.example.com//evil.test) that wasn't even in the answer key. This moves the agent from a static linter to a dynamic verifier.

Limitations and Next Steps

The benchmark is small (18 cases) and focused on Python and JavaScript single-module PRs, with a model-based judge scoring against an answer key. Ces107 acknowledges limitations, noting that the Haiku-based Proven skill still produced one false positive by misinterpreting a tested rounding behavior as a bug. Version 1.1 of the skill has already been updated to prioritize intent when tests explicitly assert behavior. The raw data, including 72 reviews and scripts, is public on GitHub for anyone to replicate.

Key Takeaways

  • Detection parity is the new baseline; stop buying skills that promise "more bugs found."
  • Trust is the new metric: demand proof (executed tests/commands) for Critical/High findings.
  • On cheaper models like Haiku, structured skills reduce noise by ~5x compared to prompt-only checklists.
  • False positives are often severity inflation; the best skills now default to lower severity when uncertain.

The Bottom Line

Proof is cheap now. Stop grading AI review tools on what they find and start grading them on what you can act on without re-checking. If it doesn't run a test, it's just guessing.