The gap between what framework documentation tells you to install and what autonomous agents actually use is wider than most vendors admit. Nolan Lawson recently published a comprehensive benchmark where he built the same Pokédex application twelve times, once for each major JavaScript framework, using a single agent per run. The results expose a critical failure mode in current AI tooling integrations: installed does not equal invoked. While costs remained relatively low, ranging from $2.67 to $4.97 per run, the real story lies in the utilization data of Model Context Protocol (MCP) servers and agent skills.

Installed Versus Invoked: The Data Gap

Lawson’s harness provided a clean environment using Podman containers with Opus 5.5 and Claude Code pinned to version 2.1.310. The critical metric wasn't just the final Lighthouse score, but the invocation rate of the specific AI tooling each framework recommends. For Nuxt, the hosted MCP server was available in every single run but was invoked in zero of them. Similarly, Next.js agents relied on skill files and AGENTS.md, ignoring the next-devtools MCP server. In stark contrast, SvelteKit, which uses a Claude Code plugin that starts its own MCP server via npx, saw the agent invoke the MCP tool in five out of five runs. This suggests that local, plugin-based integrations may currently be more reliable for autonomous agents than remote hosted servers.

The Three States of Tooling

The benchmark highlights a fundamental truth in agent evaluation: tooling exists in three distinct states—installed, loaded, and invoked. Only the final state changes the outcome of a task. Lawson argues that many current evaluations are unfalsifiable because they do not log tool calls per run. If an agent runs five times and the MCP server only fires twice, the other three runs are effectively control groups that are often misinterpreted as evidence of the tool's success or failure. The author’s own read is that the current crop of MCPs and skills may be doing more harm than good if they create noise that agents ignore or misinterpret. The data shows that skills copied into .claude/skills directories (used by Hono and React Router) had a 5/5 invocation rate, whereas hosted HTTP servers struggled to get attention.

Reproducibility Through Provenance, Not Pinning

One of the most interesting engineering choices in Lawson’s setup was leaving frameworks and their AI tooling unpinned to mimic real-world usage, where developers install the latest versions. To compensate for the lack of static pinning, the harness records detailed provenance. For npx-launched servers, the exact version is resolved and pinned at setup time. For hosted MCP servers that cannot be pinned, the system stores a SHA-256 hash of the tool list. This approach ensures that reproducibility means having a dated snapshot of the environment rather than expecting identical results from a moving target. It is a necessary shift for benchmarks in an ecosystem where model capabilities and framework APIs evolve weekly.

Key Takeaways

  • Invocation rates vary wildly: SvelteKit (plugin-based) hit 5/5, while Nuxt (hosted MCP) hit 0/5.
  • Cost is not the primary differentiator: The spread between adjacent ranks in cost ($2.67-$4.97) is within noise.
  • Code size impacts agent efficiency: Hono+htmx produced ~1,370 lines vs Angular's 2,009, affecting token costs.
  • Logging is essential: Without per-run tool call logs, you cannot distinguish between a helpful tool and an ignored one.

The Bottom Line

Stop trusting documentation that says 'install this MCP server' without verifying that agents actually call it. If you are maintaining a framework's agent integration, the metric that matters is not installation count, but invocation rate.