Your nightly summarization job missed its 06:00 deadline again. The model endpoint reported a p50 latency of 812 milliseconds—well within acceptable parameters. The dashboard flashed green. "Healthy," it said. But queue depth told a completely different story, and the job that should have wrapped up before your team arrived was still chugging along well past breakfast.

When 'Free' Costs More Than You Think

You moved the job to a free model endpoint to cut spend. Token cost dropped to zero. Sounds like a win, right? Except completion time tripled. You traded money for latency—a classic infrastructure trade-off that most dashboards don't help you see clearly. The problem isn't that free endpoints are broken; it's that "free" ignores the queue tax entirely.

Why Standard Metrics Lie to You

Traditional observability focuses on what the model is doing once it starts processing your request. P50 latency, token throughput, error rates—these all measure in-flight performance. But they completely miss the time your request spends waiting in line. A free endpoint might have decent per-request latency while burying you under queue depth during peak hours.

Measure Cost Per Completed Request

The metric that actually matters for batch workloads is cost per completed request—or more precisely, wall-clock time per unit of work completed. This combines token cost, queue wait time, and processing latency into a single number you can optimize against your SLA requirements.

Key Takeaways

  • Track queue depth alongside latency metrics—it's the hidden variable in free tier performance
  • Calculate total elapsed time from request submission to completion, not just model execution time
  • For time-sensitive jobs, a paid endpoint with predictable throughput often beats 'free' with unpredictable queues
  • Set alerts on queue depth thresholds, not just response time percentiles

The Bottom Line

"Free" isn't free when your batch job misses deadlines. Before celebrating zero token costs, measure what those requests actually cost in wall-clock time—and decide if the latency tax is worth paying. Sometimes it is. Most of the time, it's not.