If you have ever stared at a GitHub pull request comment section drowning in vague 'potential security risk' warnings, you know the pain of alert fatigue. A new tutorial from Gaurav Sharma on DEV.to breaks down how to implement AI security code reviews using MergeJudge, an open-source engine that prioritizes precision over volume. The core philosophy is simple: let deterministic rules handle the hard stops, and only let the AI raise a risk lane when it can quote the exact line from the diff.

Deterministic Rules Set the Floor

Before the AI model even wakes up, MergeJudge runs deterministic checks defined in your .mergejudge.yml file. This step catches obvious errors without burning tokens or generating noise. For example, if added lines match common secret formats like AWS keys or private key blocks, the pull request is immediately placed in the 'block' lane. Similarly, removing critical safety guards such as 'set -e' or 'persist-credentials: false' from CI configurations triggers a hard block. This ensures that low-hanging fruit is handled by simple pattern matching, leaving the AI to do what it does best.

Evidence-Gated Security Checks

The real magic happens in the optional 'Security review mode,' which is off by default. When enabled, MergeJudge scores five high-severity attack-class criteria, including 'sec.audit.injection' and 'sec.audit.authz.' However, it doesn't just guess. The system uses trust-boundary prompt discipline, instructing the model to fail a criterion only if a real trust-boundary break is introduced. Crucially, every high-severity failure must include an exact quote from the diff. If the model can't quote the line, the finding is marked as 'needs validation' rather than 'confirmed,' preventing hallucinated blocks from disrupting your workflow.

Handling False Positives and Context

MergeJudge distinguishes between confirmed findings and those requiring human validation. If the AI suspects an issue but the relevant code lives outside the diffβ€”like an auth middleware in a different fileβ€”it marks the finding with a question mark icon (❔). This keeps the pull request in the 'review' lane but doesn't block it, signaling to a human that context is missing. For larger pull requests where cross-file checks are necessary, the tutorial recommends using 'context' or 'deep' modes with your own provider key, as the free tier has strict token limits.

Key Takeaways

  • Deterministic rules for secrets and CI guards run first, blocking obvious errors before AI involvement.
  • Security review mode is opt-in and checks for injection, authz flaws, secrets/crypto issues, callback vulnerabilities, and AI tool risks.
  • AI findings are only 'confirmed' if they quote an exact line from the diff, stopping hallucinated alerts.
  • The .mergejudge.yml file is read from the base branch, preventing pull requests from disabling their own checks.
  • Use context or deep modes for complex PRs to avoid false negatives caused by limited diff visibility.

The Bottom Line

This approach is a breath of fresh air for developers tired of AI tools that cry wolf. By forcing the model to prove its case with exact quotes, MergeJudge turns security reviews from a noisy distraction into a reliable safety net. Ready to try it? Check out the full guide at mergejudge.si/guide/ai-security-code-review-github and the open-source runner on GitHub.