Unsiloed AI has announced that its Unsiloed Parser achieved a score of 88.0 on the olmOCR-Bench, placing it first among 19 evaluated services. The benchmark, maintained by Allen AI, runs 8,413 unit tests across 1,403 PDF pages to evaluate OCR performance on real-world challenges like complex tables, math equations, and degraded scans. While the headline number is impressive, the more critical takeaway for developers is Unsiloedโ€™s analysis of why traditional pass/fail metrics are becoming obsolete.

The Benchmark Landscape

The evaluation compared Unsiloed Parser against a heavy-hitting lineup including OpenAI's GPT-5.5, Anthropic's Claude Opus 4.7, and dedicated document AI platforms like Reducto and LlamaParse. Unsiloed leads the closest competitor, Nanonets OCR-3 (87.4), by a narrow 0.6-point margin. However, the gap widens significantly against general-purpose Vision-Language Models (VLMs); Unsiloed outperforms GPT-5.5 by 3.4 points and Claude Opus 4.7 by 6.1 points. The results highlight that while generalist models are improving, specialized parsing pipelines still hold an edge in structured data extraction.

Why Strict Metrics Are Broken

Unsiloedโ€™s team went beyond the leaderboard score to investigate the remaining ~12% of failures. Using an LLM-as-a-Judge protocol, they discovered that 91% of their table failures and 88% of their math failures were not actual OCR errors. Instead, these were string-level divergences that don't affect semantic meaning, such as using \dfrac instead of \frac, or minor whitespace differences. When these semantically equivalent outputs are counted as successes, Unsiloedโ€™s score jumps to 94.8%, revealing that the benchmark's rigid rules are penalizing correct outputs that simply don't match the gold reference string character-for-character.

Key Takeaways

  • Unsiloed Parser leads olmOCR-Bench with a deterministic pass-rate of 88.0, beating GPT-5.5 and Claude Opus 4.7.
  • Strict string-matching benchmarks flag ~65% of Unsiloed's 'failures' as non-issues, such as LaTeX notation variants or spacing.
  • The 'Old Scans' category remains the hardest challenge, where Unsiloed scored 52.9 on strict tests but 72.1 on semantic correctness.
  • Allen AI's olmOCR-Bench currently lacks a widely adopted metric for semantic correctness, creating a blind spot in evaluating modern AI parsers.

The Bottom Line

If you are building document pipelines, stop trusting raw pass-rates on benchmarks that don't understand LaTeX or whitespace nuances. Unsiloedโ€™s data proves that the next frontier in OCR isn't just higher accuracy, but smarter evaluation that distinguishes between a broken parser and a strict grader.