The AI industry has a scaling problem hiding in plain sight, and Modal CTO Akshat Bubna just named it. In a detailed technical breakdown, Bubna unpacked what he's calling the '100,000 sandbox problem'β€”the infrastructure bottleneck that emerges when teams try to run massive numbers of isolated inference environments for large language models simultaneously.

Why Sandboxes Are Both Necessary and Painful

LLM applications in production rarely consist of a single model answering queries in isolation. Modern AI systems typically need multiple models running concurrently, each potentially requiring separate sandboxed environments for security, resource allocation, or multi-tenant support. Bubna argues this creates a compounding challenge: as deployment scales, the overhead of managing these isolated contexts grows faster than the actual inference workload itself.

The Cost Nobody Talks About

According to Bubna's analysis, the infrastructure complexity doesn't scale linearly with model countβ€”it scales combinatorially when you factor in memory partitioning, cold start latency management, and cross-sandbox communication. Teams hitting the 100,000+ concurrent sandbox threshold find themselves spending more engineering cycles on orchestration than on actual application logic.

What This Means for Development Teams

For developers building AI-native applications, this bottleneck has real consequences: unpredictable latency spikes, ballooning cloud bills, and deployment pipelines that become increasingly fragile as they scale. Bubna suggests the industry needs better primitives for sharing model weights across inference contexts rather than treating every request as requiring full environment isolation.

Key Takeaways

  • Isolation requirements create non-linear infrastructure overhead at scale
  • Memory partitioning and cold start management compound beyond simple compute limits
  • Modal's approach targets reducing per-sandbox footprint without sacrificing security boundaries
  • Teams hitting high concurrency should question whether true sandboxing is needed for every request

The Bottom Line

Bubna's framing of this problem is a reminder that LLM infrastructure isn't just about model weights and inference speedsβ€”it's about the unglamorous work of managing thousands of concurrent execution contexts. Whoever cracks efficient multi-tenant inference at scale will have a massive operational advantage.