Stop treating HTTP 200 as proof of intelligence. A new protocol proposal from DEV.to argues that the single pass rate metric plaguing LLM evaluations is a fundamental engineering error. By collapsing reach, contract, and check into one boolean, developers are blinding themselves to whether a failure came from the network, the output format, or the model's actual reasoning. The proposed solution is a three-column scoring system that keeps these failure modes distinct, preventing a flaky server from looking like a dumb model.

The Blend Is the Bug

The core argument is that free model access exacerbates measurement noise. When a server is free, it is a second moving part that can refuse connections, queue requests, or timeout while the underlying weights remain innocent. If you score only the final checker, a network timeout and a schema violation both wear the same red mark. This makes postmortems a matter of vibes rather than data. The author notes that a missed appointment (reach failure) looks exactly like a bad heart (check failure) when you only publish the word 'healthy'.

Freeze the Exam Before You Touch the Wire

Before running any tests, the protocol demands a frozen 'exam ID'. This is generated by hashing each prompt file and then hashing the sorted list of those digests. If a prompt changes after the first request, the comparison is invalid. The checker must also be frozen, local, and pureβ€”no network calls allowed in the validation logic. The setup requires two conditions: a control transport you already trust, and the free-model call on the free server. Both must share the same timeout, written in milliseconds before the first request leaves. Editing the timeout after seeing red rows is considered cheating.

Three Columns, One Truth

The harness separates failures into three distinct stages. 'Reach' asks if the transport returned a gradable body; DNS failures, resets, and empty strings fail here and never reach the next stage. 'Contract' asks if the body matches the demanded JSON schema; a model that sounds sharp but returns prose instead of an object fails here. 'Check' is the only stage that evaluates content correctness. This ordering prevents 'fake intelligence regressions' where a flaky path is blamed on the model. The rates function in the provided Python code explicitly calculates conditional probabilities: check_given_contract and contract_given_reach, ensuring you don't advertise the survivors of a high-drop-rate server.

Key Takeaways

  • Never blend metrics: A blended pass rate is meaningless if many requests timed out. Publish the reach column separately.
  • Free tiers are unstable: Free servers introduce transport noise that can masquerade as model failure. Always pair free tests with a trusted control transport.
  • Checkers must be boring: Use local, pure functions for validation. If your checker calls an LLM, you are just building a mirror that agrees with itself.
  • Log raw bodies: When contract validation fails, capture the first 240 characters of the response. A reason code without the body is just gossip.

The Bottom Line

If you cannot write a boring, local checker, you are not ready to split a score. Stop pasting blended pass rates into your README and start publishing the specific column that moved.

Disclaimer

This protocol was prepared as part of MonkeyCode's product outreach, using their free model access and free server options as test conditions. However, the methodology is transport-agnostic. The author explicitly states that no specific quota, model list, or hardware specs were verified, and the code provided is a proposal, not an executed benchmark with sample metrics.