A subtle refactor at GoodBarber nearly cost the app platform significant compute credits, exposing a critical flaw in how job queues handle reasoning models. Pierre-Laurent Medori, backend engineering lead at GoodBarber, discovered that their system’s heartbeat mechanism failed to account for the silent thinking phase inherent to modern LLMs. By raising reasoning effort from low to medium, the time-to-first-token exceeded the 90-second Redis key expiry, causing the system to mark active jobs as dead and discard paid model calls.

The Silence That Looks Like Death

The core issue lay in the definition of "liveness." The worker refreshed its Redis heartbeat key only upon receiving tokens or moving pipeline steps. Reasoning models, however, spend significant time "thinking" before emitting the first token. During this silent stretch, the activity-driven heartbeat went stale. The system’s reaper, checking for keys older than 180 seconds, reclaimed these live jobs, treating a thinking model as a crashed worker.

A Refactor That Exposed the Hole

The crisis was precipitated by a cleanup effort that deleted 2,250 lines of legacy code, including a specific path for resuming generations after user approval. This old path contained a context manager that renewed the heartbeat every 30 seconds regardless of token activity. When the refactor consolidated all execution into the worker, it removed this safety net. Simultaneously, the team deployed an increase in reasoning effort, making the silent thinking phase longer and more likely to trigger the false-positive death signal.

The Fix: Decoupling Liveness From Output

The solution was architectural simplicity: implement a timer-based keepalive independent of model output. Medori’s team introduced a background thread that renews the Redis key every 30 seconds for every job, whether tokens arrive or not. This change ensured that a model spending two minutes on complex reasoning would not be flagged as dead. The fix was validated with a test that simulated a long wait without tokens, confirming the heartbeat persisted during silence.

Impact on Production Metrics

Post-fix data reveals a stark improvement. The rate of jobs reclaimed while pending dropped from 0.6% to under 0.1%, and the overall failure rate halved from 1.2% to 0.6%. However, the cost of reasoning remains visible: median edit times increased by 56-62% due to the higher reasoning effort. The team also noted that a separate issue—rows left pending with no subsequent activity—remained unchanged at ~0.6%, indicating a different class of failure unrelated to the heartbeat mechanism.

Key Takeaways

  • Liveness is a Timer, Not a Stream: Heartbeats must be based on time intervals, not model output events, to accommodate silent reasoning phases.
  • Audit Your Timeouts: Infrastructure components like Nginx (60s default), Cloudflare (125s default), and Redis expiries must be aligned with the maximum expected time-to-first-token.
  • Reasoning Effort is Infrastructure: Increasing model reasoning effort is not just a quality tweak; it is a load and latency change that requires infrastructure adjustments.
  • Consolidation Risks: When refactoring code paths, ensure that implicit safeguards (like the resume path’s keepalive) are explicitly migrated to the new architecture.

The Bottom Line

Silence is no longer an empty space in LLM interactions; it is a high-latency computation phase that your infrastructure must budget for. If your job queue reads silence as death, you are paying for thinking you never get to see.