A developer’s ticket-extraction pipeline appeared to achieve a stellar 99.6% success rate across 3,418 Claude API calls over two weeks. Only 14 explicit errors were logged, prompting initial pride in the system’s stability. However, this metric was a dangerous illusion. A teammate’s observation that long, complex support tickets yielded fewer action items than short, polite ones exposed the flaw: the pipeline was silently dropping data without raising alarms.

The Silent Killer: Max_Tokens and JSON_Repair

The root cause was the interaction between the stop_reason: max_tokens signal and the json_repair library. When Claude hit the token limit, it returned incomplete, malformed JSON. Instead of failing, the json_repair library automatically patched the truncated output, producing valid but incomplete JSON structures. This masked the truncation event entirely, allowing the pipeline to proceed as if the extraction was successful.

198 Hidden Failures

Upon deeper inspection, the developer discovered 198 instances where stop_reason was max_tokens. These were not logged as errors because the subsequent JSON repair made the output technically valid. This meant 198 critical data points were truncated and potentially missing key information, yet the dashboard showed near-perfect reliability. The system wasn't failing; it was quietly losing information in a way that standard error monitoring completely missed.

Key Takeaways

  • High success rates can mask silent data loss when error handling libraries like json_repair automatically fix malformed outputs.
  • Always check stop_reason independently of JSON validity. A valid JSON object does not guarantee a complete response.
  • Discrepancies in output quality based on input length are strong indicators of truncation issues.

The Bottom Line

Blind trust in automated repair tools like json_repair can turn catastrophic data loss into invisible bugs. If your pipeline doesn't explicitly log stop_reason: max_tokens, you aren't seeing your true error rateβ€”you're just seeing your ability to silently fail.