If you are picking an agent model based on the official hkust-nlp/Toolathlon leaderboard on Hugging Face, stop looking at the rank number. As of November 21, 2026, the top four models are separated by exactly 1.0 point on a 108-task benchmark. That difference is statistically equivalent to one single task flipping from fail to pass. Worse, zero of the 18 listed rows are marked as verified by the platform, and 15 of them originate from unmerged pull requests where the model publisher self-reported the score.
The Math Doesn't Support a Podium
The Toolathlon benchmark consists of 108 multi-step tasks involving 32 MCP servers and 604 tools. Because the denominator is small, each task is worth roughly 0.93 percentage points. The current #1, XiaomiMiMo/MiMo-V2.6-Pro-RL, scores 76.9, while #2 moonshotai/Kimi-K3 scores 76.5. The gap between them is 0.4 points, which is less than half of one task. When you factor in the confidence intervals inherent in such a small sample size, the ranges for the top models overlap significantly. A Wilson interval analysis suggests that 11 different rows sit within each other's plausible score range, meaning the leaderboard's strict ordering is likely noise rather than signal.
Self-Reported Data and Missing Protocols
All 18 entries on the board cite a model card as their source, meaning the vendors graded their own homework. While the benchmark paper describes a rigorous evaluation service, the board does not standardize how these numbers were generated. For instance, the GLM-5.3 card reports results averaged over three independent runs, while the Ornith-1.5-397B card cites five runs. Other entries, like Kimi-K3, do not specify the run count at all. This creates a fragmented dataset where scores are measured on different grids. Without a mandatory column for run counts, token limits, or scaffold details, comparing a 3-run average to a single-run result is an apples-to-oranges error waiting to happen.
Parameter Size Is Not a Predictor
The leaderboard also fails to correlate with model size, further complicating selection. DeepSeek-V4-Flash-Vision-Exp, with approximately 305 billion parameters, ranks fourth at 75.9. Meanwhile, the much larger DeepSeek-V4-Pro-0813, with about 1.65 trillion parameters, sits fifth at 74.1. Similarly, the 180-billion-parameter Qwen3.8-Flash-Next outranks the 2.45-trillion-parameter Qwen3.8-2.4T-A95B. Four organizationsβdeepseek-ai, ornith-ai, XiaomiMiMo, and Qwenβcontrol 10 of the 18 rows, meaning the board is largely a comparison of internal variants selected by the labs themselves, rather than an independent stress test of the broader market.
Key Takeaways
- Treat the top four rows as a single tier, not a ranking; the 1.0-point spread is within the margin of error for a 108-task set.
- Zero rows are marked 'verified' by the platform, and 15 come from open pull requests with self-reported scores.
- Inconsistent evaluation protocols (3 runs vs. 5 runs vs. unspecified) make direct score comparisons unreliable.
- Parameter count does not predict performance on this board, with smaller models often outscoring trillion-parameter giants.
The Bottom Line
Stop treating this leaderboard as a definitive ranking. Until the board enforces merged PRs, standardized run protocols, and per-task data, it is a marketing dashboard, not a rigorous benchmark.