The consensus that AI agents have overwhelmed human code review capacity is now ubiquitous, but the specific metrics backing these claims often lack transparency. While industry reports from LeadDev, Semgrep, and LinearB confirm that generated code volume has surged by 741% year-over-year, delivery speeds have only increased by roughly 30%, leaving a massive bottleneck in review. Most proposed solutions, such as risk-based routing, remain abstract concepts without reproducible data to back them up.
Testing The Risk Score Hypothesis
A new free CLI tool named hotspots attempts to bridge this gap by validating a specific heuristic: the file_risk_score. This metric combines per-file complexity and churn to predict which files are most likely to require future bug fixes. The tool's creator initially set a pre-registered correlation bar of 0.8 against future bug fix outcomes. The standalone score failed this test, achieving only a 0.515 correlation, demonstrating that raw risk scores are not direct probability predictors.
Narrower But Reproducible Findings
Despite the initial failure, a narrower insight emerged that holds up under scrutiny. Files landing in the top 10% riskiest decile of a specific diff are 6-7x more likely to need later fixes compared to the rest of the changes in that pull request. This finding survived four separate attempts to break the methodology, including moving cutoffs and removing formula terms, and successfully replicated across five repositories in five different programming languages. The full methodology, including the failed calibration test, is public.
Open Source Validation For Builders
The hotspots CLI is open-source, runs locally in under a minute, and requires no account or sales interaction. It ranks files within a current diff and flags those in the high-risk decile, turning the vague advice to review this one harder into a specific, reproducible claim. Builders can run the reproduction script on their own repositories to verify the 6-7x risk multiplier, ensuring that any risk-based routing strategy is grounded in tested data rather than vendor marketing.
Key Takeaways
- AI-generated code volume is up 741% YoY, but end-to-end delivery speed is only up ~30%, confirming the review bottleneck.
- The hotspots CLI found that top-decile risk files in a diff are 6-7x more likely to need future fixes.
- The tool's methodology is fully open source, including a failed 0.8 correlation test for the standalone risk score.
- Builders can verify the claims locally without signing up for vendor services or sales calls.
The Bottom Line
Risk-based routing is finally moving from vendor buzzword to reproducible engineering practice. If you cannot run the math on your own repo, you are not optimizing your workflowβyou are just buying into a story.