Bosco Labs, a provider of counterparty check APIs, published a benchmark testing whether LLM agents with web search capabilities can replace dedicated compliance APIs. The initial results were embarrassing for the vendor: the web-search-enabled agent using Claude Sonnet 5.5 correctly identified 23 of 25 companies, while Bosco’s own API only got 17 right. However, the follow-up analysis reveals that the web search approach is dangerously inconsistent, a finding that matters significantly for anyone building agents for regulated industries like supplier onboarding.

The First Run and Hidden Bugs

The benchmark tested 25 companies across 10 countries, including well-known entities like Apple and Siemens, as well as sanctioned or insolvent firms like Wirecard and Garantex Europe. The web arm used Anthropic’s server-side search, while the API arm made a single POST request to Bosco’s endpoint. The first run’s disparity was largely due to four bugs in Bosco’s API, such as missing historical insolvency data for Cambridge Analytica and failing to handle deleted registers for Garantex. These bugs were fixed for the second run, but the initial loss highlighted the web agent's ability to retrieve public knowledge effectively.

Inconsistency Is the Real Enemy

The second run, conducted after fixing the API bugs, yielded a score of 20 for the API and 19 for web search. This narrow margin is statistically noisy, but the variance in the web arm’s performance is the critical insight. Four companies that were correctly identified in the first run failed completely in the second, due to search limits, network errors, or the model simply not executing the search. This inconsistency makes the web-search approach unsuitable for high-stakes compliance, where an 'unknown' result caused by a failed tool call is a business risk, not just a technical glitch.

Auditable vs. Hallucinated Certainty

A deeper look at the web arm’s answers reveals a troubling pattern: in 17 of its 25 correct answers in the second run, the model stated it had not checked sanctions lists or performed no search at all. While these answers were factually correct for well-known companies like Tesco, they lack the audit trail required for compliance. The API arm, by contrast, provides specific references, such as 'UK Sanctions List RUS1421,' ensuring that every 'clean' verdict is backed by verifiable data. For agents operating in regulated environments, hallucinated certainty is worse than an explicit 'unknown'.

Key Takeaways

- Web search agents are highly variable; identical inputs can yield vastly different results due to tool failures or search limits. Dedicated APIs provide auditable, verifiable data trails, which are essential for compliance and regulatory reporting. Benchmarks run by vendors require careful scrutiny; Bosco Labs disclosed its own bugs and the limitations of their test set. For hard-to-find data (insolvency, sanctions), web search often fails to retrieve official records, whereas structured APIs excel.

The Bottom Line

Don't let a model's confidence fool you. For compliance and onboarding, the fragility of web-search agents makes dedicated APIs the only viable path for auditable, reliable decisions.