If you are building AI products today, stop obsessing over FLOPs. They are no longer the primary constraint. Michael at ArkNet Digital argues that raw compute has become the cheapest, most plentiful resource in modern AI data centers. The real bottleneck has already migrated twice—first to memory bandwidth, then to interconnects—and it is currently moving a third time, out of the chip entirely and into the power grid. For developers, this shift isn't just hardware trivia; it fundamentally dictates inference costs and determines which architectures can actually ship.

The Widening Gap Between Compute and Power

Nvidia’s latest hardware specs illustrate the crisis clearly. Comparing the GB300 NVL72 to the upcoming Vera Rubin NVL72, dense FP8 compute jumps by 3.5x, while rack power only increases by 1.4x. This discrepancy means chips are getting more capable faster than the infrastructure can feed them. The bottleneck is not a single wall but a series of walls, and the industry is walking down them one at a time. As compute scales exponentially, the linear growth of power delivery creates a hard ceiling on what a single rack can actually deliver.

Memory Bandwidth Is the Current Reality

For most developers, the pain is currently felt in memory bandwidth. During autoregressive decode, every model weight must stream from HBM to compute units. With Nvidia’s Rubin GPU, the ridge point—the balance between compute and bandwidth—sits at roughly 910 FLOPs per byte. This means that for small-batch, low-latency inference, you are bandwidth-bound by a mile. A 70B parameter model in FP8 hits a hard ceiling of about 270 tokens per second on a single GPU, regardless of what the PFLOPS slide says. Optimization strategies like quantization, continuous batching, and speculative decoding are all just ways to move fewer bytes.

Interconnects Define the Frontier

Once models and KV caches exceed a single GPU, networking becomes the problem. The NVLink domain, not the individual GPU, is now the real unit of compute. Nvidia’s Rubin NVL72 rack offers 216 TB/s of NVLink bandwidth, which is roughly seven times faster than the scale-out networking available between racks. The design rule is simple: keep chatty traffic inside the NVLink domain and only send polite traffic across the wider network. If you are renting compute, always ask for the size of the NVLink domain; two clusters with identical GPU counts can differ several-fold in useful throughput on large Mixture-of-Experts models.

The Power Grid Is the Final Boss

Zooming out, the binding constraint is megawatts, not megabytes. Nvidia’s reference design fits 40,000 Rubin GPUs into a 100 MW AI factory, resulting in roughly 2.5 kW of facility power per GPU. The IEA projects data center electricity use will nearly double by 2030, reaching 945 TWh. Meanwhile, the US grid is struggling to keep up; the median power project now takes 61 months to come online, up from 36 months in 2015. With GPUs shipping on a yearly cadence and grid power taking five years, the slowest input will inevitably set the price. Nvidia even sells 'MaxLPS' power profiles to run chips slower so more fit under a fixed power budget, signaling that power is the new scarcity.

Key Takeaways

  • Pricing features in energy is more accurate than pricing in tokens; long context and batch-1 latency are expensive because they fight bandwidth and power limits.
  • Design for latency tolerance: move workloads to async or batch processing where possible to ride the cheap side of the compute/bandwidth ridge.
  • Reduce bytes moved: Retrieval that sends 4,000 relevant tokens instead of 40,000 just-in-case tokens yields a tenfold efficiency gain that no vendor negotiation can match.
  • Budget separately for fixed capability costs and frontier capability costs, as the latter is tied to the slowest rung on the infrastructure ladder.

The Bottom Line

The last decade was a race for FLOPs; this decade is a race for watts, bytes, and wire. The winners will be the teams that move the fewest bytes per token, per joule, inside the smallest communication domain.