If you've ever stared at a PC game's system requirements and wondered whether your rig could handle it, you already understand the anxiety that LLM enthusiasts face when trying to run models locally. The question isn't just about raw horsepower—it's about one specific resource that has become the ultimate gatekeeper for local AI inference: VRAM.

Why VRAM Is Your Bottleneck

Running an LLM requires loading the entire model into memory, and unlike traditional applications that can swap data in and out of slower storage, language models need their parameters accessible at incredible speeds to maintain reasonable response times. This is why video RAM—not system RAM, not your NVMe SSD—becomes the limiting factor for most users experimenting with local deployments.

Understanding Quantization Formats

The actual VRAM requirement depends heavily on how aggressively a model has been quantized. Lower precision formats like 4-bit or 8-bit quantization dramatically reduce memory footprint, but they come with trade-offs in output quality and accuracy that vary by use case. A 7-billion parameter model might need 14GB of uncompressed FP16 memory, yet aggressive quantization could shrink that down to 4GB—making the difference between a $400 GPU and one costing five times as much. GGUF (formerly GGML) has become the standard format for consumer local inference, with several quantization levels offering different quality-to-size ratios. Q4_K_M (4-bit with optimized knowledge distillation) provides strong quality retention at roughly 60% of FP16 size—a popular sweet spot for most use cases. Q5_K_S offers higher quality but nearly doubles the file size compared to Q4 variants. For users prioritizing maximum efficiency, Q2_K can fit models on very limited hardware but introduces noticeable quality degradation in complex reasoning tasks.

GPU Recommendations by Tier

The consumer GPU market has matured significantly for AI workloads. NVIDIA dominates with its CUDA ecosystem and superior tensor core performance: Entry-Level (8-10GB): RTX 3060 (12GB variant preferred), RTX 4060, AMD RX 6600 XT—capable of running 7B models at Q4_K_M or Q5 quantization levels with room for reasonable context windows. Mid-Range (12GB): RTX 4070, RTX 4070 Ti Super, AMD RX 6800 (16GB)—opens access to 13B models at Q4_K_M and enables smoother 7B performance with higher context lengths. High-End (16-20GB): RTX 4080 Super, AMD RX 7900 XT—handles 34B quantized models at Q4_K_M or 13B at higher precision like Q8. Enthusiast (24GB+): RTX 4090 (24GB)—the consumer champion for local inference, capable of running 70B+ models at aggressive quantization or 34B models at high quality with full context windows. Apple Silicon Macs with unified memory also excel here: M3 Max configurations with 128GB unified memory can handle massive models entirely in-chip.

Quick-Reference: GPU Tier vs Model Size

The table below shows approximate VRAM requirements (in GB) for common model sizes at various quantization levels, plus typical inference overhead from frameworks like llama.cpp and Ollama. 7B Models: FP16 requires ~14GB; Q8 (~8-9GB); Q5_K_S (~6-7GB); Q4_K_M (~4.5-5GB); Q2_K (~2.5-3GB 13B Models: FP16 requires ~26GB; Q8 (~14-15GB); Q5_K_S (~11-12GB); Q4_K_M (~8-9GB); Q2_K (~5-6GB 34B Models: FP16 requires ~68GB; Q8 (~36-38GB); Q5_K_S (~28-30GB); Q4_K_M (~19-21GB); Q2_K (~12-14GB 70B Models: FP16 requires ~140GB; Q8 (~75-80GB); Q5_K_S (~58-62GB); Q4_K_M (~38-42GB) This translates to practical capability tiers: an 8GB card can run 7B at Q4/Q5 comfortably with overhead margin; a 12GB card handles 13B at Q4_K_M; a 16GB card manages 34B at aggressive quantization or 13B at high quality; 24GB opens the door to 70B at Q4.

Inference Framework Overhead

Beyond model weights, your framework consumes additional VRAM. Llama.cpp is remarkably efficient, typically requiring under 500MB of overhead for basic inference with minimal KV cache impact. Ollama adds approximately 1-2GB of overhead for its runtime management and containerization layer. vLLM's paged attention system can consume 2-4GB depending on configuration but provides significant throughput improvements for batched requests. Always budget at least 2GB of headroom beyond your calculated model requirements to account for context window expansion, KV cache allocation, and framework overhead—this prevents the out-of-memory crashes that plague local LLM experimentation.

Key Takeaways

  • Q4_K_M quantization offers the best balance of quality and accessibility for most consumer GPUs in the 8-12GB range
  • An RTX 3060 (12GB) can run 13B models at Q4_K_M; an RTX 4090 (24GB) handles 70B at aggressive quantization or 34B at high quality
  • Always reserve 2+ GB of VRAM headroom for inference overhead beyond model weights
  • Apple Silicon with unified memory provides a compelling alternative for users preferring macOS, particularly for 7B-13B workloads

The Bottom Line

Local LLM deployment has been democratized significantly—most modern gaming GPUs can handle at least 7B models competently—but matching your hardware to the right model size and quantization level still requires planning. The good news: a mid-range GPU from the past two years, combined with Q4_K_M quantization, will deliver surprisingly capable performance for the vast majority of use cases.