Running open-weights language models locally on consumer hardware has become increasingly practical, but developers frequently hit memory bandwidth walls that kill inference performance. A new deep dive into speculative decoding techniques demonstrates how to push token generation speeds up to three times faster on common GPUs like the NVIDIA RTX 4080 or 4090, as well as Apple Silicon Mac Studio systems.

What Is Speculative Decoding

Standard autoregressive inference processes tokens one at a time—the model generates each token and waits before starting work on the next. This creates a sequential bottleneck even when compute is available. Speculative decoding flips this by using a smaller "draft" model to predict multiple candidate tokens, then verifying them all in parallel with the larger target model. Accepted predictions are kept; mismatches trigger regeneration from that point.

The Hardware Constraint Problem

For models in the 27B parameter range, memory bandwidth becomes the limiting factor rather than raw compute throughput. A consumer RTX 4090 can theoretically execute thousands of teraflops, but loading model weights across the bus happens at a fraction of that speed. Developers running Mistral-7B or similar architectures locally often observe GPU utilization hovering around 30% during inference—the hardware sits idle waiting for data.

Implementation Approach

The technique requires pairing your primary model with an appropriately sized draft model—typically one-third to one-fifth the parameter count. During each decode step, the draft generates a short sequence of candidate tokens, then all candidates get validated simultaneously against the main model's next-state predictions. The acceptance rate depends heavily on how well the draft model aligns with the target distribution, which varies by use case and temperature settings.

Real-World Performance Tradeoffs

Achieving that 3x speedup isn't automatic—it requires tuning the draft model selection, sequence length, and batch sizing for your specific workload. Applications with predictable token distributions (code generation, structured outputs) tend to see higher acceptance rates than open-ended creative writing. Memory footprint doubles since you need both models loaded simultaneously.

Key Takeaways

  • Speculative decoding works best when memory bandwidth—not compute—is your bottleneck
  • Draft model quality must closely match the target distribution for high acceptance rates
  • The technique is particularly effective for repetitive, structured generation tasks
  • Both models stay resident in VRAM, so ensure you have enough GPU memory headroom

The Bottom Line

This isn't magic—it's applied systems thinking. If you're already running quantized 4-bit models and still cursing your inference latency, speculative decoding might be exactly the lever you've been missing. Test it against a representative sample of your actual workload before committing to production changes.