If your AI agent infrastructure relies solely on queue leases, you are one garbage collection pause away from sending duplicate customer emails. A new guide on DEV.to argues that standard at-least-once delivery semantics are insufficient for long-running agent jobs, where a "dead" worker can wake up just long enough to repeat irreversible side effects. The solution isn't just better timers; it's a dual-layer defense using run leases paired with fencing tokens.

The Zombie Worker Problem

The failure mode is deceptively simple. Worker A claims a run, loses network connectivity during a model call, and its lease expires. Worker B claims the run, finishes the task, and sends the output. Then Worker A wakes up, resumes from an in-memory checkpoint, and sends the output again. This isn't a bug in your agent logic; it's a feature of distributed systems. AWS explicitly notes that consumers running past visibility timeouts cause duplicate processing. For read-only tasks, this wastes tokens. For writes, it duplicates tickets, overwrites CRM fields, or spams users.

Fencing Tokens Are the Missing Layer

A lease decides who can work *now*. A fencing token decides who can *act*. The guide emphasizes that each successful claim increments a monotonically increasing fence number. If Worker A (fence 17) tries to commit an action after Worker B (fence 18) has taken over, the downstream system must reject the stale request. This comparison must happen at the side-effect boundaryβ€”like an email service or database write proxyβ€”not just in the orchestrator. A stale worker can hold valid credentials and a network connection; only the fence proves it is outdated.

Implementation: Atomic Claims and Conservative Renewals

To make this work, you need atomic claims. The article provides a PostgreSQL pattern where the UPDATE statement increments the fence, sets the new lease owner, and returns the new state in a single transaction. If no row returns, the worker does not proceed. Renewals must also be conditional: a worker can only extend its lease if it still matches the current lease_owner and fence. If renewal fails, the worker must immediately enter "lost ownership mode," stopping new tool calls and discarding local state. Do not try one last commit; if you can't prove ownership, you lose the right to act.

Key Takeaways

  • Leases are not enough: They protect the scheduler but not downstream APIs from stale workers.
  • Fence at the boundary: The service performing the side effect (e.g., email, DB write) must validate the fence number.
  • Set lease duration by p99 checkpoints: Don't set leases for the whole run time; set them for the interval between durable checkpoints.
  • Test ugly cases: Simulate network partitions and expired leases to ensure stale fences are rejected before production traffic hits.

The Bottom Line

Stop treating agent runs as simple function calls. They are distributed processes that require strict ownership protocols. If your architecture doesn't have a way to reject a stale worker's write based on a generation number, you are building a system that is guaranteed to duplicate side effects.