The most expensive incident in your agent infrastructure often starts with a green dashboard. Requests return, workers hum, and the queue just keeps getting longer until a slow model call turns that wait line into timeouts, duplicate work, and a frozen UI. The missing piece isn't another worker; it is AI agent backpressure, a explicit contract that stops the system from accepting more work than it can finish within a useful deadline.

Why A Queue Is Not Backpressure

A queue is merely storage between producers and consumers. Without backpressure, an API accepts every request, agents fan out into numerous model and tool calls, and workers retry endlessly when a provider slows down. The system appears available until most requests have waited too long to matter. As the source notes, "A long queue is not resilience; it is delayed rejection." This is critical for agents because one customer request can trigger many downstream calls, holding concurrency for far longer than a standard API call.

Define The Overload Contract

Stop letting implementation details decide how your system behaves under pressure. You must define what each work class does when capacity is low. Interactive chat should reject or degrade quickly with a 429 or 503, while background tasks should queue within hard bounds and return a job ID. Bulk work can be deferred, and critical repairs need reserved slots. The key question is simple: can this request still produce value if it waits? If not, a quick rejection is kinder than an infinite spinner.

Bound Every Accumulation Point

Capping only your job broker is a rookie mistake. Work accumulates in HTTP connection pools, in-process semaphores, model providers, and database connections. You need limits at four scopes: global concurrency for shared resources, per-tenant concurrency to stop noisy neighbors, per-workflow concurrency to prevent fan-out explosions, and per-dependency concurrency to isolate slow tools. Keep large payloads out of the queue; store object references instead. A bounded queue must contain enough info to restart work, not unlimited data that bloats memory while waiting.

Fairness Requires Scheduling, Not Hope

FIFO is vulnerable. A single tenant dumping thousands of background jobs can starve a newly arrived interactive task. Implement priority bands with round-robin selection within each band. Reserve capacity explicitly: try 60% for interactive work, 25% for normal background, 10% for critical repairs, and 5% for bulk work. Add ageing so long-waiting background jobs eventually get a boost. Do not expose precise queue positions unless your scheduler guarantees them, because retries and priority changes make exact numbers untrustworthy.

Retries Must Yield To Pressure

Retries are the ultimate overload amplifier. A timeout doesn't prove the provider missed the request, and immediate retries steal capacity from new work. Treat retry eligibility as policy, not a catch block. Record the dependency, idempotency key, and deadline for each attempt. Honor Retry-After headers, use exponential backoff with jitter, and stop when the run deadline passes. Wrap the entire call in a semaphore so unlimited retries cannot bypass your concurrency limits. For side-effecting tools, pair this with idempotency keys to avoid duplicate writes.

Stream Honestly, Monitor The Bad Promise

Streaming can hide overload. If a chat connection stays open while the job waits in a worker queue, the user thinks the model is thinking. Send state transitions like queued, running, or waiting_for_dependency instead. Monitor signals that reveal a bad promise: queue depth, oldest queue age, and time from admission to first useful output. Alert on combinations, not single numbers. Rising queue depth plus rising oldest age means demand exceeds service. Flat depth plus rising latency points to a downstream dependency.

Key Takeaways

  • Backpressure is an application decision to queue, defer, degrade, or reject work before it piles up.
  • Separate rate limiting (frequency) from admission control (current capacity) and budgeting (cost).
  • Never silently downgrade a request that changes safety or truthfulness; state what context was available.
  • Start rollout in observe-only mode to compare theoretical admission against real queue age and completion data.

The Bottom Line

Stop praying your queue drains. Implement explicit admission control that rejects work when capacity is gone, or you are just building a better way to disappoint users slowly.