The dream of escaping API bills by running Large Language Models locally is technically feasible but brutally constrained by hardware physics. As detailed in a new deep-dive from Internals Decoded, local inference replaces the abstracted model-as-a-service with direct computation on your CPU, GPU, and memory. The tools are mature—Ollama and llama.cpp are the standard stacks—but the performance ceiling is hard-coded by your hardware’s memory bandwidth. You aren't just swapping endpoints; you are inheriting the entire burden of memory hierarchy management.

The Bandwidth Bottleneck

Speed in local inference is determined by a single, unforgiving equation: decode tokens per second roughly equals memory bandwidth divided by the bytes read per token. This means your GPU's VRAM speed matters more than its core count. For instance, an RTX 5090 with 1,792 GB/s of bandwidth can hit roughly 290 to 300 tokens per second on a 7B model, while an M2 Max with 400 GB/s tops out around 66 tokens per second. If your model weights spill into system RAM, performance plummets because system memory bandwidth is a fraction of VRAM's. Hybrid mode, where layers are split between GPU and CPU, is a compromise that often results in sluggish generation for interactive chat.

VRAM and the KV Cache Trap

Capacity is the other silent killer. While quantization shrinks model weights—Q4_K_M averages about 4.9 bits per weight, requiring roughly 0.6 GB per billion parameters—the KV cache grows linearly with context length. For Llama 3.1 8B, a 128K context window demands 16 GiB of VRAM just for the cache, on top of the weights. Many local setups fail here: the model loads fine, but the first long prompt runs out of memory. The solution is asymmetric KV cache quantization, such as keeping keys at 8-bit and values at 4-bit, which can reduce cache memory by 59% with minimal perplexity loss.

When Local Makes Sense

Local inference is not a universal win. It changes the trust boundary, ensuring prompts never leave your machine, which is critical for regulated workloads. It also shifts costs from per-token fees to upfront hardware and electricity. However, if you need high concurrency, a single workstation GPU running Ollama will struggle. Production-grade local servers require engines like vLLM or TensorRT-LLM, which use continuous batching and paged attention to squeeze throughput from the same hardware. For a single user on a laptop, Ollama is convenient; for a team, it is often insufficient without significant engineering effort.

Key Takeaways

  • Local inference speed is primarily limited by memory bandwidth, not CPU/GPU core count, making high-bandwidth VRAM critical for real-time interaction.
  • The KV cache grows linearly with context length and can consume more VRAM than the model weights themselves, causing out-of-memory errors during long chats.
  • Asymmetric quantization (8-bit keys, 4-bit values) reduces KV cache memory usage by approximately 59% with negligible impact on model perplexity.
  • While Ollama is ideal for single-user convenience, production environments requiring high concurrency should utilize dedicated serving engines like vLLM or TensorRT-LLM.

The Bottom Line

Local inference trades financial predictability for data sovereignty, but it demands a sophisticated understanding of memory bandwidth and cache management to avoid unusable latency.