Cloudflare’s newly trending security-audit-skill, which racked up nearly 2,000 GitHub stars in a single day, passed all four security findings against a 47-line Node service, even though two were false and one cited a non-existent file. This incident highlights a critical distinction in AI security tooling: the validator checks for schema compliance, not truth. The skill, now used as the starting point for Cloudflare’s fleet-wide vulnerability harness, relies on a multi-agent architecture where hunter agents generate candidates and verifier agents attempt to refute them. While the validator confirmed the format of the false positives, it was the manual application of the skill’s own skeptical methodology that correctly identified the errors.

The Validator Checks Shape, Not Substance

The skill’s validator, validate-findings.cjs, enforces strict formatting rules such as relative path safety and severity ordering but never opens source files to verify claims. In the test case, a finding claiming SQL injection in a non-existent file src/users.js was marked valid because it adhered to the JSON schema. Similarly, a false positive in the getNote function passed validation because the cited line existed and contained a query, even though the parameters were safely bound. The validator’s design philosophy, explicitly stated in Phase 4 of the skill, is that success proves "format and ledger consistency only," not the absence of bugs.

Skepticism Is the Expensive Part

Finding candidates is cheap; disproving them is where the value lies. Cloudflare’s internal data shows that without a dedicated refutation step, early validation rejection rates can hit 40%, flooding maintainers with bad tickets. The skill addresses this by allocating two of its six phases solely to disproof, with Phase 3 employing fresh agents to rebuild controls and reproduce results. In the 47-line test, manually walking through this refutation prompt revealed that the getNote payload returned null, disproving the SQL injection claim. This step, which requires reading the code and running the payload, is what separates a useful report from noise.

Cost and Independence Challenges Remain

Despite its architectural strengths, the skill faces practical hurdles. Users report high token costs, with one Hacker News commenter noting 150k tokens for a small FastAPI project. A blind test cited in GitHub Issue #20 showed the skill’s "quick profile" costing a median $29.95 compared to $2.06 for a plain agent, with no significant gain in bug recall on clean targets. Furthermore, the skill’s verifiers often inherit the same model as the hunters, lacking the true independence found in Cloudflare’s production pipeline, which uses different models for triage and hunting. Fresh context is not the same as a different mind.

Key Takeaways

  • The Cloudflare security-audit-skill validator passes false positives if they match the schema, requiring human or independent agent review for truth.
  • The skill’s core value lies in its Phase 3 refutation process, which manually disproves findings by rebuilding controls and running payloads.
  • High token costs and potential model dependency between hunter and verifier agents remain significant limitations for small teams.
  • Cloudflare’s internal data suggests a 40% rejection rate without proper refutation, emphasizing the need for skeptical verification steps.

The Bottom Line

Stop treating AI security validators as truth engines; they are format checkers. The real product is the skeptical workflow that kills false positives, and if your tool doesn’t have a robust, independent refutation step, it’s just generating expensive noise.