Every LLM project eventually hits the same wall: your eval suite and unit tests demand a live model, and CI immediately slows to a crawl. Costs balloon, secret keys become mandatory dependencies, and tests flake whenever a vendor quietly updates a checkpoint. The output is nondeterministic by design, meaning a "did the pipeline work" test can fail for reasons entirely unrelated to your code changes. The solution is a deterministic mock provider that implements the exact same interface as your real adapters but runs completely offline.

The Interchangeability Requirement

The entire technique depends on one critical architectural decision: your real provider and your mock must be perfectly interchangeable. You need a narrow interface that every adapter conforms to, covering request normalization and response shaping. The response object must carry latency, token usage, and cost fields, not just the output string. If your mock only returns text, every code path that reads usage or costUsd stays untested. Model the full shape or accept blind spots in your telemetry.

Determinism Through Seeded Hashing

The core promise is that the same input always yields the same output. The implementation derives everything from a hash of the prompt, feeding that seed into a small pseudo-random generator. FNV-1a combined with mulberry32 provides zero-dependency determinism. Because the seed comes from the prompt, even the fake latency is reproducible. Run the suite a thousand times and you get identical numbers, which is what makes snapshot-style assertions possible without network variance.

Steerable Outputs and Degrade Modes

A useful mock is not random; it is steerable. Two levers control the output: fixtures for exact canned responses matched as case-insensitive substrings, and directives embedded in the prompt itself. Directives like "echo," "json," or "reverse" let example suites exercise real scorer behavior without per-test wiring. But the feature that earns its keep is the degrade mode. A green test suite proves your scorers accept good output but never proves they reject bad output. A deterministic corruption function that breaks exact match while preserving partial similarity proves your scorers actually discriminate.

Key Takeaways

  • Mocks must implement the full response shape including latency, usage, and cost to avoid untested code paths
  • Seeded hashing with FNV-1a and mulberry32 provides zero-dependency determinism for reproducible fake latency
  • Directive-based prompts enable scorer testing without per-test fixture wiring
  • Degrade modes are essential to verify scorers reject bad output, not just accept good output

The Bottom Line

Deterministic mocks and real model evals are complementary, not interchangeable. Run the mock on every commit to protect your plumbing, and run real model evals on a schedule to measure quality. Confusing the two is how teams end up with a perfectly green suite and a product that gives terrible answers.