If you are building agents and relying on the Hugging Face SWE-Bench Pro leaderboard for ranking, you are comparing apples to oranges. As of October 31, 2026, the board lists 47 rows, none of which are marked verified. The top score is 94.94, but the second place sits at 68.74. This 26.2-point cliff isn't necessarily a sign of a massive performance gap; it is likely a sign that the board is mixing two different benchmark versions without clear labels.

The Version Confusion Problem

SWE-Bench Pro now exists in two distinct configurations. V1 contains 731 tasks, while V2, released on September 22, 2026, contains 642 tasks after 89 were dropped for invalid tests and 480 were repaired or retired. The board's official task ID points to V2, yet the top row, PCL-RSI-lab/Seshat-agent, explicitly reports its score on V1 (694/731). The evidence dataset for this row confirms the IDs match the V1 list exactly. While the math for 94.94% is correct, it is a V1 number displayed on a V2 board. This means the gap between first and second place conflates model capability, agent harness differences, and benchmark version changes.

Missing Verification and Harness Details

The board lacks basic documentation standards for agentic coding. Of the 47 rows, 41 do not state which agent harness was used. In agentic workflows, the scaffold is as critical as the model; a score without a harness is effectively a score for an unknown system. Furthermore, only one row provides a full per-task evidence dataset. The rest rely on model cards or links to Scale's public leaderboard, which are outdated for older models. The board does not specify if runs were single-attempt or multi-attempt, nor does it confirm if network access was restricted, a key variable in the V2 protocol that locks network access to the model endpoint only.

Statistical Noise and Data Integrity Issues

Even if we ignore the version mismatch, the leaderboard struggles with statistical resolution. With 731 tasks, the standard error is significant. Twelve rows between ranks 4 and 15 fall within a 7.1-point band, which is roughly the size of a single unpaired detectable gap. This means the rankings in the middle of the pack are likely noise. Additionally, there are data integrity issues: one quantized checkpoint, INCModel3/GLM-5.1-MXFP4-Mixed-CT-AutoRound, lists a score of 58.4 copied directly from its base model's card, with no evidence it was actually re-evaluated. Another row, MuVeraAI/Laguna-XS.2, cites a source that disagrees with its own listed score by 1.8 points.

Key Takeaways

  • The Hugging Face SWE-Bench Pro board mixes V1 (731 tasks) and V2 (642 tasks) scores without explicit version tags in the row format.
  • The top score of 94.94 is a V1 result, while the board's default configuration is V2, creating an apples-to-oranges comparison with the second-place score.
  • 41 of 47 rows fail to list the agent harness used, making it impossible to attribute performance to the model alone.
  • Zero rows are marked as verified by the platform, and two rows appear to copy scores from base models or other repositories without independent validation.
  • Statistical noise at this task count means gaps under 5 points are likely insignificant, rendering the middle-tier rankings unreliable for decision-making.

The Bottom Line

Treat the current SWE-Bench Pro leaderboard as a collection of self-reported claims rather than a ranked truth. Until Hugging Face enforces version tagging, harness disclosure, and verification, you cannot use this board to objectively compare agent systems. If you are building in production, run your own evals on the specific V2 config with a locked network protocol. Don't let a 26-point cliff convince you of a quality gap that is actually just a version mismatch.