Developer AziizBg has released "passproof," an npm package that addresses one of the most insidious reliability issues in AI agent pipelines: silent test failures masquerading as successes. The tool, available via npx passproof install, enforces a simple but critical constraintβ€”an AI agent cannot claim tests passed unless the runner actually printed the output.

Why This Matters for Agent Reliability

Modern AI agents are increasingly tasked with running test suites and reporting results autonomously. But here's the problem: most CI systems and test runners have exit codes that can be misleading. An agent might receive a "success" signal while tests silently failed in a background process, or the output got swallowed by buffering issues. Passproof cuts through this ambiguity by requiring visual confirmation of test execution before any assertions about success are made.

The Core Mechanism

The package works by wrapping your existing test runner and capturing its stdout/stderr streams directly. It then parses that output to determine actual pass/fail status, rather than relying solely on exit codes or agent interpretation. When integrated into an agent's workflow, passproof acts as a trust layerβ€”it won't allow the pipeline to proceed until it has verified the raw output matches the claimed result.

Real-World Implications

This is particularly relevant for agents running in environments where test output might be truncated, redirected, or lost entirely. Docker containers, remote execution environments, and certain CI configurations have all been documented sources of "ghost failures"β€”tests that failed but whose error messages never reached the calling agent.

Installation and Usage

Getting started is straightforward: npx passproof install adds the package to your project. From there, you configure it to wrap your existing test command (jest, vitest, pytest, etc.), and passproof handles the verification layer automatically. The tool outputs structured JSON indicating whether tests genuinely passed based on captured output.

Key Takeaways

  • AI agents frequently misreport test results due to relying on exit codes instead of actual output
  • Passproof forces visual verification before allowing success claims
  • Works with any test runner that produces stdout/stderr output
  • Solves "ghost failure" issues in containerized and remote execution environments

The Bottom Line

This is exactly the kind of unsexy infrastructure work that makes AI agents actually reliable instead of just sounding impressive. Passproof won't make headlines, but it'll stop your agent from lying to you about whether tests really passedβ€”and that's worth its weight in debugging hours saved.