A model that doesn't know your library doesn't fail by doing nothing. It fails by confidently writing code for a library it does know. This is the core thesis of a new deep-dive by Kevin CC, maintainer of the testing library TWD. When an AI agent encounters an internal API client or a niche open-source package, it fills every gap with the closest known analog—importing from the popular alternative, assuming return shapes, and inventing option names. The result is code that looks right, compiles in the agent's head, and is subtly wrong in production.

The Knowledge Layer Problem

The first line of defense is what you feed the model. Kevin argues that whatever you provide—whether it's an AGENTS.md file, a skill definition, or an llms.txt—is the only place the model can learn your API. Instead of filling this context with warning labels like "never do X," which prose cannot enforce, he recommends full signatures, explicit return types, and complete working examples. One correct example beats three paragraphs of explanation. Crucially, he notes that context isn't free. His initial 32 KB reference doc forced the agent to load everything for trivial tasks. The solution was a layered approach: an 8 KB core for common tasks, with detailed references loaded on demand via Claude Code's references/ directory support.

Determinism Over Prose

If the model hallucinates, you need a mechanism to catch it. Kevin advocates for making the compiler teach the API. By strictly typing public interfaces, you ensure that misspelled options or incorrect argument orders become compile errors rather than silent bugs. He also implemented a focused linter with four specific rules to catch errors types miss, such as missing await keywords or imports from the wrong library. The key to making this work with agents is zero configuration and autofix capabilities. If an agent has to configure a linter, it won't run it reliably. Furthermore, error messages must be actionable; instead of "invalid usage," the linter tells the agent exactly what to do next, turning the error log into a self-correcting loop.

Evaluating the Agent

Instructions regress like any code. A minor edit can silently break a rule the agent depends on. Kevin uses evals to test his agent instructions, running them via claude plugin eval. These evals use deterministic graders—like regex checks on generated files or verifying if a specific tool was called—rather than relying solely on LLM judges. He found that prior to fixing his instructions, an eval case scored 0.50, meaning half the checks failed on a task he believed was covered. These evals cost money (about $1.80 for five cases), so he runs them before releases, while using a free, offline claude plugin validate --strict on every PR.

Key Takeaways

  • Layered Context: Split documentation into a small core and on-demand references to save tokens and reduce noise.
  • Type Safety as Documentation: Strict typing forces the agent to adhere to the actual API surface, catching hallucinations at compile time.
  • Actionable Linting: Use zero-config linters with clear, fix-oriented error messages to guide agent self-correction.
  • Deterministic Evals: Test agent instructions with regex and tool-usage checks, not just LLM judgments, to catch regressions reliably.

The Bottom Line

Until models have perfect knowledge of every niche library, developers must engineer determinism into the agent workflow. Prose warnings are useless against confident hallucinations; strict types, layered docs, and automated evals are the only things that actually keep agents honest.