If you're building fintech tooling that uses LLMs to extract structured JSON from code review diffs, you've probably hit a wall: the documents are too long, the model times out, or you blow past token limits entirely. A new technical breakdown on DEV.to spells out a pragmatic architecture for handling exactly this scenarioβ€”and it starts with abandoning the "one giant request" mental model.

The Core Problem With Long-Context Extraction

Code review history grows unboundedly. A single PR might touch hundreds of files, and your extraction pipeline needs to pull structured findings from all of it. But throwing an entire change history at one LLM call is a recipe for intermittent failures. Models hallucinate when context windows get cramped near their limits, response times become unpredictable, and debugging becomes nightmare fuel because you can't isolate which part of the input caused the failure.

Chunk Everything With Hard Boundaries

The solution outlined in the piece: put a token budget AND a deadline around every document chunk. That means splitting your long documents into manageable pieces before they ever reach the model, with explicit limits on both tokens-per-chunk and wall-clock time per request. If a chunk exceeds its budget or times out, you fail fast and can retry just that slice rather than reprocessing everything.

Retrieve Evidence Per Finding Type

Instead of dumping all context into one massive prompt, the architecture retrieves evidence specifically for each finding type you're trying to extract. The LLM gets targeted context about security issues separately from performance concerns, separately from compliance checks. This dramatically reduces token consumption per call while improving output quality because the model isn't wading through irrelevant noise.

Merge Validated JSON in a Worker

The final piece is a dedicated worker that merges validated JSON fragments and records tenant ownership throughout the pipeline. Each chunk's output gets validated independently before merging, so a malformed extraction from one section doesn't corrupt your entire dataset. The worker maintains clear attribution back to which tenant owns each findingβ€”a non-negotiable requirement for fintech compliance.

Key Takeaways

  • Never make one model request responsible for reading an entire change history
  • Enforce both token budgets and deadlines per chunk, not just one or the other
  • Retrieve evidence per finding type rather than dumping all context together
  • Validate JSON fragments independently before merging in a centralized worker

The Bottom Line

This isn't rocket science, but the pattern is clearly being reinvented by teams that skip architecture upfront. If you're doing LLM-based extraction on long documents and haven't implemented chunk-level budgets with isolated validation, you're one bad PR away from production headaches.