When a network connection dies after a deployment provider accepts a release request, most automation flows treat the missing response as a failure. They retry the write. This is a dangerous mistake. The release may be running, it may have failed before starting, or the provider may have accepted the request but delayed execution. From the caller's perspective, these three outcomes are indistinguishable, yet treating them all as failures turns reliable systems into duplicate-action machines.

Idempotency Is Necessary But Insufficient

Idempotency keys prevent the same logical request from creating two effects, provided the entire path respects that key. However, they do not answer whether the first operation committed or what state it left behind. These questions are often collapsed into a single boolean, but they should remain separate. A local service might remember dispatching key 'deploy-42', but the provider may not expose a lookup by that key. Or the caller might time out before persisting the operation ID. In these cases, replaying the local record is not reconciliation; it only proves what the caller remembers, not what actually happened externally.

Model Attempts and Outcomes Separately

To fix this, keep two distinct records. The attempt record describes the requested external action, including the stable operation ID, idempotency key, actor, target, payload hash, and provider route. The outcome record describes what can actually be proved, such as provider acknowledgement, object ID, current state, and independent verification. This split prevents an accepted HTTP request or an in-memory success flag from masquerading as the business outcome. For deployments, 'command started' is not the same as 'revision serving healthy traffic.' For payments, a client timeout says nothing reliable about whether money moved.

Give Uncertainty a Durable State

Your state machine needs a place where automation stops making new writes. A practical sequence moves from 'prepared' to 'dispatching' to 'acknowledged' to 'verifying', but must branch to 'unknown' when evidence is missing. This 'unknown' state should be persisted, not thrown as a transient exception. It must carry context for another process to continue investigation, storing the request fingerprint, last confirmed transition, missing evidence, and reconciliation owner. This turns vague errors into bounded operational tasks. Many provider APIs cannot prove a global negative, so when coverage is incomplete, the honest result is still unknown, requiring manual review rather than manufactured certainty.

Reconcile With Reads Before Writes

Safe recovery is deliberately boring. Freeze redispatch, keep the original idempotency key, and mark the operation as unresolved. Query the provider using a request ID or deterministic name, then match the intended effect by comparing target, payload hash, and timestamps. Verify the customer-visible result through an independent read, as provider acknowledgement alone may be insufficient. Only permit a new write after confirmed absence, using a new attempt identity while retaining the link to the unresolved attempt. Tests must inject uncertainty at boundaries, such as timeouts after commits or crashes before persisting IDs, to ensure at most one external effect exists.

Key Takeaways

  • Idempotency keys prevent duplicates but do not prove external state changes occurred.
  • Separate attempt records from outcome records to avoid confusing requests with results.
  • Persist an 'unknown' state to block blind retries and enable structured reconciliation.
  • Use authoritative reads to verify effects before considering any new write operations.
  • Manual review is safer than assuming certainty when provider APIs cannot prove absence.

The Bottom Line

Stop treating timeouts as failures. Add an 'unknown' state to your domain model and define one authoritative read path before adding another retry policy. It makes your system less eager and much easier to trust.