Demilade Ayeku’s latest build, Pinned, is a version-aware code reviewer for Sanity that refuses to let LLMs guess at API behavior. Submitted for the Sanity Challenge Path One, the agent parses client configs and GROQ queries to tell developers exactly what their code does today, what breaks if they bump the apiVersion, and where Sanity’s own documentation contradicts itself. This isn’t just another chatbot wrapper; it’s a deterministic engine backed by a Knowledge Base that treats version boundaries as structured data rather than ambiguous prompts.

The Drafts Leak and the Version Boundary

The core motivation is a subtle bug lurking in countless 2024 tutorials. Before API version 2025-02-19, the default perspective was 'raw', meaning authenticated queries returned unpublished drafts alongside published content. This leaked draft data to production sites. Bumping the version flips the default to 'published', stopping the leak but silently breaking preview code that relied on the drafts. Pinned detects this specific conjunction: a token present, apiVersion before 2025-02-19, and perspective unset. It flags the risk without requiring the model to understand the nuance of 'both true at different times'.

Why Deterministic Rules Beat Pure LLMs

Ayeu’s evaluation reveals why pure LLM approaches fail here. In a three-arm test over 15 configs, the model alone achieved 80% precision but missed citations. Adding a Knowledge Base improved recall to 82% and citation accuracy to 100%, but precision plummeted to 45% as the model hallucinated conflicts that didn’t exist. Pinned’s hybrid approach—using a Babel AST parser to extract typed facts and a deterministic rule engine to evaluate them—achieved 100% precision, 100% recall, and 15/15 exact matches. The model is only used for explanation, never for decision-making.

Knowledge Base as Conflict Resolution

The agent uses Sanity Context’s Knowledge Base to handle documentation inconsistencies. Instead of asking 'which doc is right?', Pinned uses Knowledge Base Instructions to recognize version-scoped truths. For example, it knows 'default perspective is raw' and 'default perspective is published' are both correct, separated by the 2025-02-19 boundary. This structure allowed the build to self-identify three critical conflicts in Sanity’s docs, including a changelog error where the function name sanity::partOfRelease mismatched its example partOfReleases. The agent cites these findings with verbatim quotes and retrieval dates, ensuring every claim is traceable.

Key Takeaways

  • Pinned uses a Next.js 16 stack with Vercel AI SDK 7 and @ai-sdk/mcp to separate rule evaluation (deterministic) from explanation (LLM).
  • The agent’s rule engine achieves 100% precision and recall on its eval set by parsing code into an AST rather than relying on keyword search.
  • Sanity’s own documentation contained three verifiable errors or ambiguities that the Knowledge Base surfaced automatically.
  • The project is built with Claude Code, with a public session available for those wanting to see the agent’s reasoning process.

The Bottom Line

Pinned proves that for versioned APIs, 'ground truth' is a moving target. By treating version boundaries as structured data rather than asking an LLM to resolve conflicts, Ayeku built a tool that actually understands the fine print. This is the future of dev-tool agents: deterministic engines for logic, LLMs for language.