For teams desperate to keep proprietary data off third-party servers, the question isn't which model is smartest, but which GPU has the VRAM to run it. Frank, founder of Berlin-based engineering firm web3.berlin, recently published a practical guide for sizing local inference hardware, arguing that every private AI deployment comes down to a single bottleneck: GPU memory. His rules of thumb, derived from testing models like Qwen and DeepSeek on a lab rig of four RTX 3090s, offer a sobering correction to the 'buy the biggest card' instinct.

The 0.6 GB Per Billion Rule

Frank's primary heuristic for model weights is starkly simple: at 4-bit quantization (the Q4_K_M format common in GGUF files), expect to need roughly 0.6 GB of VRAM per billion parameters. This accounts for the sub-5-bit average per weight in that quantization tier. The practical implications are immediate: a 7-8B model fits in 5-6 GB, a 14B model needs 9-10 GB, and a 32B model consumes around 20 GB. That 32B class is the sweet spot for most companies, but it already maxes out a single 24 GB card like an RTX 3090. A 70B model requires 40-45 GB, forcing a multi-GPU setup before you've even loaded a document.

The KV Cache Tax Nobody Budgets For

The weights are only half the story. Frank emphasizes that the KV cache, which stores attention states for every token in your context window, adds a second layer of VRAM consumption that scales linearly with context length. Using the formula 2 ร— layers ร— kv_heads ร— head_dim ร— bytes_per_value, he demonstrates that Qwen2.5-32B at FP16 requires about 256 KB per token. A 32,768-token context thus demands roughly 8 GB on top of the 20 GB for weights, blowing past a 24 GB card entirely. Models using grouped-query attention, which have fewer KV heads, are substantially cheaper here, a critical consideration when selecting open-weight models for long-document RAG pipelines.

Three Deployment Archetypes

Frank categorizes private AI hardware into three tiers. A single 24 GB GPU workstation handles up to 32B models for small teams doing document search and basic assistant tasks. Multi-GPU workstations with 48-96 GB VRAM (two to four cards) accommodate 70B-class models or multiple concurrent services, with llama.cpp's --tensor-split flag handling the layer distribution. For GDPR-sensitive clients who want EU data sovereignty without office hardware, Frank recommends dedicated GPU servers in EU data centers, citing Hetzner in Finland as an example. His own lab runs four RTX 3090s, a Threadripper 3960X, and 256 GB of DDR4, using llama.cpp and Docling for production document processing.

Key Takeaways

  • Budget 0.6 GB VRAM per billion parameters at 4-bit quantization for model weights alone.
  • KV cache adds gigabytes for long contexts; Qwen2.5-32B needs ~8 GB for a 32k context window.
  • 32B models are the current sweet spot for enterprise RAG, but they exceed single 24 GB card capacity with long contexts.
  • Multi-GPU setups require attention to power (1.4 kW for four RTX 3090s), heat, and noise, not just VRAM.
  • EU data center GPU servers offer a GDPR-compliant alternative to office hardware for regulated industries.

The Bottom Line

Frank's math is a useful reality check: local LLM deployment is a VRAM budgeting exercise disguised as a hardware purchase. The 0.6 GB per billion parameter rule is a solid starting point, but the KV cache tax and power/cooling overhead are where amateur sizing projects quietly fail.