The first time an LLM feature hits real traffic, the logs fill with 429s and somebody writes a retry loop. It usually looks like the one from the last REST API they integrated: catch the error, sleep, try again, maybe double the sleep. It ships, the errors drop, and everyone moves on. Then the bill arrives, or a batch job takes four times longer than it should, or the app keeps hammering the API for an hour after the account ran out of budget. The retry loop was never the hard part. Rate limits on LLM APIs behave differently from a typical REST API, and handling them well is mostly about not retrying when it won't help, and not sending the request in the first place when you already know it will fail.

Stop Stacking Your Retries

Most official LLM SDKs ship with built-in retries. The Anthropic TypeScript SDK, for example, retries connection errors, 408, 409, 429, and 5xx responses twice by default with a short exponential backoff. This covers transient blips, including the 529 overloaded_error the API returns under heavy load. The critical mistake is wrapping that built-in logic in your own retry loop. Three attempts from your code times three attempts from the SDK results in nine requests per call. Under real load, this turns a brief slowdown into a stampede. Pick one layer to own retries. For interactive requests, the SDK default is usually fine, but for background jobs, raise the limit per request instead of writing a wrapper.

Not Every 429 Is a Rate Limit

This is the trap. On the Claude API, a 429 with error type rate_limit_error can mean two very different things. You might have exceeded a per-minute limit for requests or tokens, in which case the response carries a retry-after header and waiting works. Alternatively, your organization might have hit its monthly spend cap. This has the same status code and error type but lacks a retry-after header, and the error details specify enforced_spend_limit_reached. Access only returns at the start of the next month or when the cap is raised. Every retry, including the SDK's automatic ones, fails. A naive loop treats both the same, burning through retries and logs on a request that cannot succeed. Classify the error before you retry: stop for spend limits, and retry-later for transient capacity issues.

Read the Headers Before You Hit the Wall

Retries are reactive, but LLM APIs also provide proactive signals. The Claude API returns headers such as anthropic-ratelimit-requests-remaining and anthropic-ratelimit-input-tokens-remaining on every successful response. The API uses a token bucket, meaning capacity refills continuously rather than resetting on the minute. A limit of 60 requests per minute can be enforced as roughly one per second, meaning bursts get rejected even when the per-minute total looks fine. Furthermore, input tokens read from the prompt cache do not count toward the input-tokens-per-minute limit. If you send the same large system prompt on every call, caching it is often the cheapest way to raise effective throughput. You can read these headers to throttle a concurrency limiter, shifting from discovering the limit by failing to slowing down as you approach it.

Key Takeaways

  • Avoid stacking retry loops: SDKs already retry transient errors (429, 5xx) by default, so wrapping them in your own logic multiplies requests and causes stampedes.
  • Distinguish error types: A 429 with 'enforced_spend_limit_reached' requires stopping the worker and alerting a human, whereas a transient rate limit allows for retry-after logic.
  • Use response headers for throttling: Monitor headers like 'anthropic-ratelimit-input-tokens-remaining' to proactively adjust concurrency before hitting hard limits.
  • Leverage prompt caching: Input tokens read from the cache do not count toward input-token rate limits, offering a cheap way to increase effective throughput.

The Bottom Line

Treat rate limits as a capacity-planning signal, not an error to swallow. Sometimes the right fix is not code at all: batch APIs, prompt caching, or a smaller model for easy requests can remove the pressure entirely. Read the provider's rate-limits page end to end, because the edge cases live in the footnotes.