Running local LLMs is less about magic and more about math, specifically the unforgiving arithmetic of memory allocation. A new analysis published on DEV.to by bestaitools-fr breaks down the precise formula used by the osFoundry desktop application to determine if a model will actually fit on your hardware. The core equation is straightforward but critical: need = weights + KV cache + overhead, which must remain less than or equal to 80% of your usable memory to ensure stability.

The Weight of Quantization

The largest component of memory usage is the model weights, which scale directly with parameter count and quantization level. The article provides a specific table of bytes per weight for common formats, noting that Q4_K_M requires approximately 0.56 bytes per parameter due to block scaling overheads, rather than the theoretical 0.5. For a 14B parameter model in Q4_K_M, this results in roughly 7.3 GiB just for the weights, a figure verified against osFoundry’s display for a MacBook M3 with 16GB of RAM.

The Hidden Cost of Context

Many users underestimate the KV cache, which grows linearly with context length. Using Qwen2.5 14B as a case study, the article calculates that a 4K context window adds 768 MiB to the memory footprint, while a 32K context balloons this to 6 GiB. This means a model that comfortably fits in VRAM at short contexts can completely fail to load when you attempt to utilize longer context windows, making context length a primary variable in local inference planning.

Usable Memory vs. Total Memory

The distinction between total RAM and usable memory is where most local setups fail. For dedicated GPUs, VRAM is the hard ceiling regardless of system RAM. On Apple Silicon, despite unified memory architecture, the system reserves resources, so 75% of total RAM is a realistic working figure for the GPU. The analysis suggests that if your calculated need exceeds 80% of this usable memory, you are entering the danger zone where the OS may swap to disk, rendering inference unusably slow.

Bandwidth Dictates Speed

Memory size doesn’t just determine if a model loads; it determines how fast it runs. The article explains that generation speed is capped by memory bandwidth divided by model size. A base M3 chip offers roughly 100 GB/s bandwidth, meaning an 8.4 GB model (like a 14B Q4_K_M) will max out around 12 tokens per second, while a smaller 8B model might hit 20 tokens per second. This relationship holds true in practice, linking the feasibility check directly to performance expectations.

Key Takeaways

  • Q4_K_M quantization uses ~0.56 bytes per parameter, not 0.5, due to scaling overheads.
  • KV cache grows linearly with context; a 32K context can add 6 GiB to a 14B model.
  • On Apple Silicon, assume only 75% of total RAM is available for GPU inference.
  • Keep total memory need under 80% of usable RAM to avoid disk swapping.
  • Inference speed is inversely proportional to model size given fixed bandwidth.

The Bottom Line

Stop guessing if your Mac can run a model; calculate the weights, add the context tax, and check your bandwidth. If the math doesn't work, no amount of hope will make it run at acceptable speeds.