If you've ever gotten a Slack message from finance asking why your OpenAI bill quadrupled overnight, you know the panic sets in fast. Your gateway was running fine. Users were happy. Nobody touched anything. Yet somehow you're staring at a invoice that looks like it belongs to a much bigger company. These aren't bugs or crashes โ€” they're cost traps. Defaults that make perfect sense when you're testing locally but become financial landmines once traffic scales up. After running LiteLLM Proxy in production across three companies, I've personally walked into every single one of these. Today I'm sharing the fixes so you don't have to learn them the expensive way (literally).

Trap #1: The Retry Spiral โ€” When num_retries=3 Actually Means 15

Here's how it works: You've configured a fallback chain of five models with three retries each on your LiteLLM proxy. A request fails, gets retried, hits a fallback model, that fails too, and the cycle continues. The math gets ugly fast. One user request can trigger up to twelve upstream API calls โ€” and you pay for every single one, including the ones that errored out after consuming tokens. If your expensive model is GPT-4o and each retry chews through 2K input tokens before timing out, a single failed request could cost you more than twenty-four thousand tokens. The fix isn't disabling retries โ€” it's capping them at the chain level rather than per-model. Set max_fallbacks to two as a hard cap on chain depth, keep allowed_fails low so your circuit breaker trips before you're bankrupt, and crucially: make sure your fallback model is cheaper than your primary. If GPT-4o fails, fall back to GPT-4o-mini โ€” not Claude Opus.

Trap #2: Fallback Chains That Funnel Money Into Premium Models

This one cost me two thousand three hundred dollars in a single weekend. A well-meaning engineer set up fallbacks that looked logical on paper: gpt-4o-mini falls back to gpt-4o, which falls back to claude-3-5-sonnet, which falls back to claude-3-opus. The reasoning was sound โ€” if the cheap model fails, try something better. But during a traffic spike, GPT-4o-mini hit OpenAI's TPM limits and started returning 429s everywhere. Every request fell through the entire chain to Claude 3.5 Sonnet. For six hours straight, we were running one hundred percent of our traffic on the most expensive models available. Rate limits are per-model, not per-gateway โ€” so if a spike is caused by overall volume rather than a model-specific outage, your fallback just hits the same limit on different hardware while charging you ten times more. The solution: structure fallbacks by cost tier, never capability tier. When GPT-4o-mini fails, fall back to gemini-1.5-flash or claude-3-haiku โ€” other cheap models in the same price bracket. If all cheap options fail, return an error rather than escalating to premium tiers. Set cooldown_time to sixty seconds so you're not hammering the same failing model repeatedly.

Trap #3: Zero Caching Means Paying for the Same Answer a Thousand Times

Most teams don't enable LiteLLM's built-in Redis caching because they assume their prompts are too dynamic. But I audited one team's production traffic and found that thirty-four percent of all requests were exact duplicates of queries made in the previous hour. They were paying OpenAI roughly four hundred dollars per day for identical completions. The documentation buries caching under Advanced Settings, so it stays disabled by default while teams focus on getting their gateway working. Enabling Redis caching with a one-hour TTL took about thirty seconds and cut that team's daily spend from four hundred to one hundred eighty dollars โ€” a fifty-five percent reduction. For providers that support it like Claude and GPT-4o, also enable prompt caching which charges ninety percent less for cached input tokens.

Trap #4: No Per-Key Budget Limits Means One Runaway Loop Bankrupts You

An intern once pushed a while True loop to staging. It didn't crash โ€” it just called the gateway four thousand times per minute with a four kilotoken prompt. By the time PagerDuty fired, eight hundred forty-seven dollars had been spent in twelve minutes. LiteLLM's max_budget field exists but most teams never configure it during initial setup. The fix is setting budgets at three levels: global caps as your emergency brake (like five hundred dollars per day), team-level limits via virtual keys, and individual developer budgets when generating API credentials. With a fifty-dollar daily limit on the intern's key and a rate limit of one hundred requests per minute, that runaway loop would have been throttled after roughly one hundred calls and blocked entirely once it hit fifty dollars in charges โ€” total damage around eighty cents.

Trap #5: The Streaming Tax โ€” Paying for Tokens You Never See

Streaming is great for user experience. Users see tokens appear in real-time instead of waiting for the full response. But here's what most teams miss: when a streaming request gets interrupted mid-stream because a user navigates away or their connection drops, you still pay for the entire generation. I've seen teams where twenty-three percent of their token spend was on completions no user ever actually saw because the client disconnected early. LiteLLM doesn't automatically cancel the upstream request when the client disconnects โ€” it's happily receiving tokens from OpenAI and forwarding them to nobody. Enable streaming_client_disconnect in your config so LiteLLM closes the upstream connection when it detects a broken client stream, and add conservative max_tokens caps on streaming endpoints as an additional safeguard.

Key Takeaways

  • Cap total retry attempts across your entire fallback chain, not just per-model โ€” num_retries multiplied by fallback_depth is your real exposure
  • Structure fallbacks by cost tier: cheap to cheap, mid-tier to mid-tier. Never let requests escalate from budget models to premium ones
  • Enable Redis caching today. If you can't answer whether it's enabled in five seconds, check โ€” thirty-four percent duplicate traffic is common
  • Set per-key budgets before shipping. Global caps, team limits, and individual developer quotas create defense in depth against runaway processes
  • Track abandoned streaming requests with custom callbacks. Twenty-three percent waste on undelivered tokens is fixable

The Bottom Line

Every one of these traps is a sensible default that becomes dangerous at scale โ€” retries multiply, fallbacks cascade, caching stays optional until it's eating thirty percent of your bill. The pattern holds: don't disable the feature, add constraints instead. Budgets, caps, cooldowns, TTLs. Your gateway should work for you, not drain you while you're not looking.