If your AI agent ships a patch with 100% line coverage, you might still be shipping untested garbage. The metric that matters isn't how many lines executed, but which specific test would fail if that code broke. A new methodology detailed on DEV.to exposes the flaw in file-level coverage by introducing a hunk ownership map, classifying diff hunks as OWNED, INCIDENTAL, or ORPHAN to reveal the true safety net of agent-generated code.

The Illusion of Green Checks

Consider an agent that adds a retry wrapper and touches an old helper. Line coverage looks pristine because a pre-existing smoke test executes the new lines by accident. However, that old test never asserts anything about retries. In this scenario, lines 52-55 might have no covering test at all, while only lines 60-63 are protected by a new test introduced by the patch. File-level metrics collapse these critical distinctions into a single misleading percentage, hiding the fact that most of the new logic is effectively unverified.

Building the Hunk Ownership Map

The proposed solution leverages coverage.py contexts to map per-test execution data directly onto diff hunks. By running pytest with --cov-context=test and generating a JSON report, developers can parse git diff -U0 to identify added-line ranges precisely. A Python script then joins these inputs to classify every hunk: ORPHAN hunks have no covering test, INCIDENTAL hunks are covered by pre-existing tests that don't validate the new behavior, and OWNED hunks are protected by tests added or changed by the patch itself. This granular visibility forces a review of which code is actually guarded by assertions.

Auditing Fixtures and Stability

Not all OWNED hunks are created equal. The methodology demands a fixture reach audit, using pytest --fixtures-per-test to identify shared fixtures that might make unrelated tests execute code paths accidentally. If a patch modifies a fixture reachable by most of the suite, it’s a signal to review the change as a test infrastructure modification. Furthermore, stability is a precondition: owners must run covering files repeatedly with fixed shuffle seeds. If a test verdict flips, it’s noise, not evidence, and the hunk should be demoted to ORPHAN until the flake is fixed.

Key Takeaways

  • File-level coverage is insufficient for agent patches; it masks untested logic behind incidental execution.
  • Classify hunks as OWNED, INCIDENTAL, or ORPHAN to determine if a test actually validates new behavior.
  • Use coverage.py contexts and git diff -U0 to map test execution to specific added-line ranges.
  • Audit fixture reach to detect accidental coverage and enforce stability preconditions to ignore flaky tests.
  • Block INCIDENTAL and ORPHAN hunks in CI gates to ensure only OWNED code merges without human waiver.

The Bottom Line

Stop letting agents hide behind green checkmarks. If a test doesn't explicitly assert the new behavior, it doesn't own the code. Adopt the hunk ownership map to expose the gap between executed lines and verified logic.