The scenario is all too familiar: It is Tuesday, and the ML lead demands 8 A100s by Friday. You hit launch, get InsufficientInstanceCapacity, and watch a quota request sit in 'under review' while the one region with capacity charges a 40% premium. GPU infrastructure on the cloud breaks the muscle memory you built on CPU VMs. The hardware is scarce, on-demand pricing is brutal, and the default 'just autoscale it' advice fails because the unit of scarcity is the accelerator, not the CPU cycle.

Inference and Training Are Opposites

You need to treat these as two distinct workloads with conflicting requirements. Inference demands low latency, high uptime, and autoscaling based on queue depth. Training craves raw throughput, fast inter-node networking, and tolerates interruption if it can checkpoint. The core design principle is keeping the accelerator busy. For inference, requests must be batched; running 16 requests in one GPU pass is significantly cheaper than 16 separate passes. For training, the bottleneck is rarely computeβ€”it is often the network fabric synchronizing gradients across nodes.

Pick the Right Silicon and Pricing Model

Stop reaching for the biggest GPU available. Size by VRAM first: if your model weights and activation memory need 60GB, a 24GB card is useless regardless of price. Use inference-class GPUs like NVIDIA L4, A10G, or T4 for serving small to mid-size models; they offer the best latency-per-dollar. Reserve training-class monsters like A100s and H100s for fine-tuning or training where large VRAM and NVLink interconnects are mandatory. For pricing, match interruptibility to the job. Use on-demand for production serving, spot instances for checkpointed training (saving 60-90%), and reserved capacity only for steady, predictable baselines.

The Autoscaling Trap: Scale on GPU, Not CPU

The most common engineering mistake is setting up a HorizontalPodAutoscaler (HPA) that watches CPU metrics. A GPU pegged at 100% utilization often shows near-idle CPU usage, meaning your autoscaler never triggers, and your endpoint falls over under load. You must scale on a GPU-aware custom metric, such as gpu_duty_cycle or request-queue depth. Behind the scenes, configure your GPU node pool to scale to zero when no pods are pending. A dev endpoint left running over a weekend is a five-figure surprise, not a rounding error.

Storage, Networking, and Quota Reality

Never bake model weights into your container image. It bloats pull times and couples model versioning to build pipelines. Store artifacts in object storage (S3/GCS) with versioned paths and load them once at startup. For multi-node training, co-location is critical. Use cluster placement groups and high-bandwidth fabrics like AWS EFA or InfiniBand to keep nodes physically close; cross-zone chatter quietly halves throughput. Finally, file quota requests the moment a project hits the roadmap. GPU quotas start at zero, and human review can take days. If you wait for the 'Friday deadline' to ask, you are already late.

Key Takeaways

  • Inference and training require opposite infra strategies; do not mix them.
  • Always size by VRAM first; a cheap card that cannot load the model is worthless.
  • Autoscale on GPU utilization or queue depth, never on CPU metrics.
  • Use spot instances for training to save 60-90%; never for stateful serving.
  • Request GPU quotas weeks in advance, not days before launch.
  • Scale GPU node pools to zero when idle to prevent massive weekend bills.

The Bottom Line

GPU cost discipline is not about cutting corners; it is about respecting the scarcity of the hardware. If you are paying for idle accelerators, you are failing at basic engineering hygiene. Right-size your instances, automate your scaling based on actual GPU pressure, and treat quota management as a critical path item, not an afterthought.