Recent initiatives by Anthropic and GitHub Next have pushed AI-assisted formal verification into the spotlight, promising code that is mathematically proven correct. However, a new analysis from Keller AI warns that these proofs only establish correctness relative to a specific specification and a set of explicit assumptions. The core issue for builders is that a verified proof is only as good as the model behind it, and AI agents are currently best at constructing the proof while humans must still validate the premise.

The State of AI Formal Verification

Anthropic’s collaboration with Theorem demonstrated the scalability of this approach, verifying 239 entry points across 123 libraries. Claude constructed proofs that were then checked by the Rocq proof assistant, establishing crash-freedom for inputs under 64 MB. Similarly, GitHub Next and Microsoft Research are exploring the PDD Kit, which uses Verus to check Rust implementations against specifications. These tools allow agents to search for arguments, but a deterministic checker ultimately determines if the steps are justified, separating the creative search from the logical validation.

Specifications Are the New Bottleneck

The article highlights a critical distinction between testing and proving. While testing a sorting function on [4, 2, 4, 1] might show it works, it doesn’t prove it works for every input. Formal verification uses induction to cover all cases, but this relies entirely on the specification. If the business requirement is "deny access immediately after revocation," but the technical specification says "allow access if the session has permission," the code can be mathematically proven correct while still failing the business logic. Developers must now spend more time reviewing these specifications to ensure they accurately express intent.

Hidden Assumptions and System Boundaries

Proofs also depend on assumptions about the external environment, such as RISC-V, Linux, and memory behavior in the Anthropic study. A proof might assume that an API contract is truthful, but if the integration returns a cached decision from before a revocation, the proof’s premise no longer describes reality. Furthermore, formal methods often miss side-channel attacks. A token comparison might return the correct boolean but leak information through execution time. A proof about return values alone does not acquire coverage for timing properties unless the model explicitly includes them.

Key Takeaways

  • AI agents like Claude can construct complex proofs, but deterministic checkers (Rocq, Verus) are required to validate them.
  • Verification is conditional: a proof establishes correctness only within the bounds of the written specification and stated assumptions.
  • Specification review is becoming a primary human task as agents handle more implementation details.
  • Side-channel vulnerabilities and integration errors can exist even in formally verified code if the model is incomplete.

The Bottom Line

Formal verification is a powerful tool for infrastructure, but it is not a silver bullet. It shifts the burden of correctness from testing code to verifying specifications, requiring developers to be as rigorous with their definitions as they are with their implementations.