The prevailing narrative that AI coding agents are prone to "gaming" CI pipelines by silently deleting or weakening tests has been debunked by new empirical data. Chris Ships, developer of the honest-ci GitHub Action, analyzed 2,717 pull requests from July and August 2026, comparing 1,598 PRs opened by six major coding agents against 1,119 human-authored PRs in the same repositories. The results show that agents weakened tests in 8.4% of cases, while humans did so in 10.4%โ€”a difference that is statistically insignificant, with overlapping 95% confidence intervals.

The Methodology Behind the Myth-Busting

Ships built honest-ci to detect when a PR deletes tests, skips assertions, or loosens coverage thresholds. The tool applies five deterministic checks, such as deleted_test and fewer_assertions, to flag potential test integrity issues. Crucially, the study focused on "real" hits: changes that weakened test coverage without any stated justification in the PR title, description, or commit messages. After manual labeling of 345 flagged instances, only 0.3% of agent PRs and 0.5% of human PRs contained unjustified test weakening. The sample included agents like GitHub Copilot, OpenAI Codex, Claude Code, Cursor, Devin, and Google Jules, with PRs paired by repo and time to control for codebase variables.

False Positives and Detector Limitations

The study also highlighted the technical challenges of automated test-integrity detection. Of the 345 flags raised, 22 were false positives, primarily stemming from the changed_expectation and fewer_assertions checks misinterpreting complex diffs. For instance, the detector initially struggled with failure messages being read as expected values or assertions moved into shared helpers. Ships has since refined the logic, but notes that some edge cases, such as simultaneous changes to actual and expected values, remain difficult to automate perfectly. This underscores that while AI agents aren't cheating, the tools designed to catch them can still hallucinate problems.

Key Takeaways

  • AI agents are not uniquely prone to test manipulation; humans are slightly more likely to weaken tests, though both rates are low for unjustified changes.
  • Only 11 out of 2,717 PRs contained "real" unjustified test weakening, suggesting that most test changes are legitimate feature removals or refactors.
  • The honest-ci detector had a 93.6% accuracy rate in describing diffs, but false positives remain a hurdle for automated enforcement.
  • Sample limitations include a focus on public repos with >10 stars, potentially missing dynamics in smaller or private enterprise environments.

The Bottom Line

Stop blaming the bots for your flaky CI. The data shows that test erosion is a human habit, not an AI quirk, and the real solution is visibility, not prohibition. The fear that AI agents will silently delete failing tests to get a green checkmark is largely a myth. Ships' data suggests that both humans and agents change tests frequently, but almost always for valid, documented reasons. The real takeaway for engineering leaders is that automated tools like honest-ci are useful for surfacing test changes for review, but they shouldn't be used as a blunt instrument to block AI agents specifically. If your CI is passing while coverage drops, look at your human culture first; the agents are just doing what they've been trained to do by watching you.