Most engineering teams build LLM integration layers with a naive assumption: try Provider A, and if it throws an exception, switch to Provider B. This model, often implemented as a simple retry loop, collapses under production load. A recent deep dive by developer ariesrandy on DEV.to argues that routing is not a safety net but a complex, stateful execution system that must account for asynchronous signals, billing coordination, and divergent provider behaviors.
The Myth of the Linear Request
The standard mental model of input โ provider โ output fails because generation requests are not atomic function calls. In reality, a single request involves authentication, schema validation, health-based reordering, credit reservation, and state persistence before a single token is generated. The author notes that routing is computed before execution begins, using logic like reorderByHealth to dynamically adjust provider sequences based on real-time status. This means the 'first' provider is never fixed, and the system must be designed for dynamic, not static, execution paths.
Storage Is the Source of Truth
Providers disagree on what 'success' means. One may return a 200 OK while the output file is still processing; another may timeout but deliver results via webhook later. To handle this, the system treats storage and the database as the ground truth rather than trusting immediate API responses. A critical pattern introduced is the 'grace period'โa deliberate sleep followed by a check of storage files. If files exist, the provider succeeded despite the API signal. Failing over without this check leads to double execution, overwritten valid outputs, and inconsistent state transitions.
Cost and Failover Are Tightly Coupled
Routing decisions are financial decisions. The system must reserve credits before execution and resolve outcomes as 'used,' 'failed,' or 'cancelled' after. Failover is not just a switch; it involves cancelling the current prediction, logging the cost outcome, and marking a generation_failover_in_progress flag in the database. This flag is essential to prevent webhook handlers from prematurely refunding credits or marking jobs as failed while a secondary provider is still attempting to complete the task. Without this coordination, cost leaks accumulate across retries.
Providers Are Fundamentally Different Systems
Switching endpoints is not enough when providers have different execution models. Black Forest Labs uses async jobs with polling and signed URLs; BytePlus uses synchronous responses with manual finalization; Cloudflare offers edge-based sync execution. Even basic concepts like 'completion' vary. The author argues that the real abstraction is not the provider API, but a unified state machine exposing generic fields like generation_status and generation_output_file. The system must map multiple inconsistent lifecycles into this single abstraction.
Key Takeaways
- Routing is proactive: It is computed based on health and capability before execution starts, not just triggered by failure.
- Trust state, not signals: Providers are unreliable; storage and database convergence are the only valid sources of truth for success.
- Failover requires coordination: Cancelling predictions and managing credit reservations are part of the execution flow, not post-hoc cleanup.
- Abstract the lifecycle: Build a state machine that normalizes async polling, webhooks, and sync responses into a single user-facing status.
The Bottom Line
If your routing logic looks like a try-catch block, you are building a liability. Production-grade LLM infrastructure requires a stateful execution engine that treats failure as a normal state, not an exception.
Practical Implementation Notes
For builders implementing this, the author suggests exposing a unified interface that hides provider-specific schemas. Instead of returning raw provider data, return normalized fields like generation_id and generation_provider_order. This allows the backend to handle the messy reality of provider inconsistencies while giving the frontend a clean, deterministic view of the job's progress. Stop treating routing as a fallback mechanism and start treating it as the core definition of your execution under uncertainty.