Building your own measurement harness is a rite of passage for performance engineers, but it is also a trap. Developer Marvin Oka recently detailed a painful lesson learned from building a custom harness: he found ten bugs in it. Every single one of those bugs made the results look better than they were, and critically, not one was discovered by simply reading the code. This highlights a dangerous blind spot in self-hosted performance infrastructure where logic errors silently inflate metrics.

The Illusion of Correctness

When you write a harness, you are often optimizing for ease of use or specific project constraints, not necessarily for rigorous scientific validation. Oka’s experience suggests that the bugs were not obvious syntax errors or crashes, but logical flaws that biased the output. These 'silent killers' are the most dangerous kind of bug in performance testing because they don’t throw exceptions; they just lie to you. If your benchmark looks too good to be true, it probably is.

A Practical Checklist for Validation

In response to this debugging nightmare, Oka published a six-point checklist designed to catch these subtle errors. The core philosophy of the checklist is utility over aesthetics: it is meant to be used rather than admired. Each check includes a definition of what it proves and, equally important, what it does not prove. This nuance is vital for builders who need to know the limits of their validation methods, preventing false confidence in the remaining unknowns.

Why Code Review Fails Here

The fact that none of the ten bugs were found by reading the code is a stark reminder of the limitations of static analysis and peer review for measurement logic. Code review is excellent for catching architectural issues, security flaws, or obvious logic errors. However, it struggles with complex state management, timing issues, or edge cases in data aggregation that only manifest under specific load conditions. You cannot review your way out of a bias; you have to test the tester.

Key Takeaways

  • Self-built harnesses are prone to optimistic bias, where bugs inadvertently improve reported metrics.
  • Code review is insufficient for validating measurement logic; active testing of the harness itself is required.
  • Validation checklists should explicitly state what they do not prove to avoid false confidence.
  • Ten bugs were found in a single custom harness, all of which skewed results favorably.

The Bottom Line

If you are building your own measurement infrastructure, assume it is lying to you until proven otherwise. The only way to trust a number from a custom harness is to break the harness itself.