Every few weeks a new benchmark says one local inference stack is X% faster than another. The numbers are usually real. They're also usually answering a different question than the one you have. Here is how to choose, by workload.

The Setup Tax and Single-User Experience

For solo developers engaged in exploratory model testing, Ollama remains the pragmatic choice. If you are pulling a new model every few days to see how it behaves, setup time is the bottleneck, not tokens per second. Ollama’s one-command pull and sensible defaults win here, and a slower decode rarely matters when you are reading the output yourself. One catch: set the context length explicitly. If you compare models on defaults, you are not comparing like with like.

Maximizing Hardware Efficiency on Fixed Resources

When the hardware is fixedβ€”such as a single consumer GPU, a laptop with no discrete GPU, or Apple Siliconβ€”and you want every token per second, build llama.cpp yourself with the backend flags for your hardware. Generic prebuilt binaries often take conservative paths. You also get direct control over layer offloading, KV cache quantization, and batch sizes.

Concurrency and the Shift to Shared Infrastructure

The calculus changes entirely when multiple users access the system simultaneously. The moment several requests arrive concurrently, the question changes from how fast is one answer to how many answers per GPU-hour. Continuous batching lets finished requests leave and new ones join mid-batch, and paged KV cache keeps memory from fragmenting. Single-user benchmarks do not show this at all.

How to Run a Fair Comparison

To ensure your selection is based on reality rather than noise, adhere to these strict comparison protocols: use the same model file and quantization, set the same context length explicitly, and use identical prompts including one long prompt for prefill and one long answer for decode. Measure time-to-first-token and decode tokens per second separately, and test at both concurrency 1 and your real expected concurrency. Finally, write down the driver, backend, and build flags used.

Key Takeaways

  • Ollama optimizes developer time and setup friction, ideal for exploratory work.
  • llama.cpp optimizes hardware utilization on fixed, single-machine setups.
  • vLLM optimizes throughput and memory management for concurrent, multi-user loads.
  • Fair comparisons require identical model files, quantization levels, and explicit context lengths.
  • Metrics must separate time-to-first-token from decode tokens per second to be meaningful.

The Bottom Line

Ollama optimizes your time, llama.cpp optimizes one machine, and vLLM optimizes a shared GPU. Pick the one that matches what you are actually short of.