CI pipelines are drowning in false positives, but a new proposal argues that boolean job statuses are insufficient for handling them. A technical note published on DEV.to outlines a strict 'flake freeze citation rule' that requires two distinct witnessesβ€”a local runner and a remote replayβ€”to share specific cryptographic identifiers before a flake can be frozen. The core thesis is simple: if you can't prove the failure is stable across different execution environments using identical inputs, you don't have a flake; you have a bug or a configuration error.

The Three-Part Identity Check

The proposed gate rejects any freeze request unless two witnesses share a fixture pin, a property digest, and a canonical bound. The fixture pin is a SHA-256 hash of the test data prefix and bytes, ensuring the environment hasn't drifted. The property digest hashes the checked-in specification file, preventing model drafts or uncommitted changes from validating a freeze. Finally, the bound must match exactly. If a patch rewrites the fixture bytes, the pin changes, and the gate refuses to cite a freeze. This forces developers to treat fixture changes as code changes subject to review, rather than silent environmental noise.

Local Authority, Remote Evidence

A critical constraint in this rule is the distinction between local and remote roles. The decision function explicitly rejects any batch where a remote witness attempts to set freeze_writer to true. Only the local caller has the authority to mint a freeze citation. Remote jobs are permitted to append witnesses to the ledger, but they cannot authorize the freeze itself. This prevents scenarios where a misconfigured remote agent accidentally suppresses a real defect by writing a freeze flag. If the remote evidence is missing or single-sourced, the gate returns a refusal, demanding that developers do not invent the missing role.

Signature Stability Over Boolean Pass/Fail

The rule demands that all failing witnesses within the scoped group share a single, stable failure signature, such as a specific exception class or assertion ID. A change from AssertionError to KeyError is treated as a different failure mode, not flicker. The proposal includes a Python reference implementation using standard library dataclasses, which checks for these conditions before returning an admit decision. If the signatures differ, the system assumes multiple bugs are present and rejects the freeze. This moves the conversation from 'did it fail?' to 'did it fail in the exact same way on different machines?'

Key Takeaways

  • No Boolean Shortcuts: A simple pass/fail bit is rejected; witnesses must carry cryptographic hashes for fixtures and specs.
  • Authority Separation: Remote runners can only append evidence; they are explicitly forbidden from writing freeze flags.
  • Signature Consistency: All failures in a freeze scope must share an identical error signature; mixed errors indicate multiple bugs.
  • Strict Rejection: The gate uses a first-match-wins filter for rejections, prioritizing malformed data and authority violations over partial matches.

The Bottom Line

This is a necessary correction to the lazy engineering of CI flakes. By forcing cryptographic identity and separating authority, we stop hiding real bugs behind the 'flaky test' excuse. If your team can't meet these strict witness requirements, you don't have a flake problemβ€”you have a tooling problem.