Most AI agent demos operate in a sterile vacuum where tool calls return clean 200s and the agent moves on. In production, however, network hiccups cause timeouts that leave the agent blind to the actual state of the transaction. A new analysis on DEV.to argues that these retry bugs are not failures of large language model reasoning, but rather classic distributed-systems errors exacerbated by the addition of an LLM layer.

The Illusion of Failure

Consider a standard support ticket scenario where an agent decides to issue a refund. It calls POST /refunds, the payment service processes the transaction and writes to its ledger, but the network drops the response packet. The agent's HTTP client hits a timeout and raises an error, interpreting the lack of response as a failed action. The agent then retries, resulting in a duplicate refund. The model's reasoning was perfect; it selected the correct tool and arguments. The failure occurred entirely in the execution layer, after the decision was made.

The Three-State Trap

Standard agent frameworks often collapse distinct network states into a single "tool failed" signal. This is dangerous because a timeout indicates only that an answer wasn't received, not that the action didn't happen. There are three distinct scenarios: the request never reached the service, the service rejected the request, or the service succeeded but the response was lost. Only the first two are safe for blind retries. Treating the third scenario as a failure leads to double-execution of state-changing actions.

Idempotency and Durable State

The solution lies in rigorous application-level controls rather than better prompting. Idempotency keys must be generated by the application before the first attempt and reused on every retry, ensuring that a second call is a no-op. Furthermore, the record of intent cannot live solely in the conversation context window, which is volatile. It must be written to durable storage before execution. Without this, a crash or context truncation erases the memory of the attempted action, leading to inconsistent states.

The Missing "Unknown" State

Most action models rely on a binary terminal state of success or failure. Reliable systems require a fourth state: "unknown." This state should trigger reconciliationβ€”querying the downstream system by order ID or idempotency keyβ€”rather than automatic retries. When an action lands in "unknown," the agent must block further attempts and escalate to a background job or human review. This separates the model's proposal from the application's execution truth.

Key Takeaways

  • Timeouts are ambiguous; they do not confirm that a state-changing action failed.
  • Idempotency keys must be generated by the application, not the LLM, to prevent double-execution.
  • Agent intent must be persisted in durable storage before execution to survive context window resets.
  • Implement an "outcome unknown" state to block automatic retries and trigger reconciliation workflows.

The Bottom Line

Stop treating LLMs as reliable network protocols. The fix for agent flakiness isn't a smarter model, it's boring, robust distributed-systems engineering like idempotency keys and durable intent logs.