Speculative decoding just got cheaper for local Gemma 4 users. A new repository by developer xbill9 introduces exact 4-bit repacks of Google's Multi-Token Prediction (MTP) drafters, shrinking the file size from 97.8 MB to 59.2 MB for the E2B model. This isn't just about disk space; it translates to a significant reduction in GPU memory overhead, dropping the drafter's cost from 120 MiB to 80 MiB on an RTX 3050. For builders running local LLMs on constrained hardware, this is a massive win for context window headroom.

How The Repack Works

Google ships its QAT drafters in bf16, but the weights are already quantized to a 4-bit grid. Standard llama.cpp converters default to Q8_0, introducing slight rounding errors and bloating the file. The gguf_exact.py script identifies the original trained steps and stores the weights as Q4_0 without re-rounding. The result is a file that is 39% smaller than the Q8_0 version and 3.6x closer to Google's original weights by relative error. Since the drafter shares the target model's KV cache, this smaller footprint directly frees up VRAM for longer contexts.

Integration With llama.cpp

llama.cpp now supports Gemma 4's drafter via the --spec-type draft-mtp flag. The drafter is a lightweight 4-layer model that reads the target's hidden state rather than maintaining its own cache. To use the new exact files, you download the q4_0-exact.gguf from Hugging Face and point the -md flag to it. The configuration requires --ngld all to offload the drafter to the GPU and --spec-draft-n-max to set the guess limit. On the RTX 3050, the sweet spot for most tasks was two drafted tokens per step, yielding an average of 2.68 tokens per pass in coding tests.

Performance and Limitations

While the memory savings are concrete, the speedup depends heavily on the workload. Predictable outputs like code see high acceptance rates, with one test showing 160 of 189 guesses accepted. However, free-form stories result in more rejected drafts, which wastes compute cycles. The gain also diminishes under heavy multi-user load because the GPU loses the spare capacity needed to verify speculative tokens. Additionally, greedy text generation can shift at near-ties due to batched arithmetic rounding, though the output remains valid for the target model.

Key Takeaways

  • The exact Q4_0 repack reduces Gemma 4 E2B drafter size to 59.2 MB and VRAM usage to +80 MiB.
  • Accuracy is higher than llama.cpp's default Q8_0 conversion, with 3.6x lower relative error.
  • Use --spec-type draft-mtp in llama.cpp; start with --spec-draft-n-max 2 for best results.
  • Speculative decoding benefits single-user, memory-bound scenarios but may degrade under high concurrency.

The Bottom Line

If you are running Gemma 4 QAT targets on local hardware, the exact Q4_0 repack is the obvious choice. It offers the smallest file size, the lowest memory cost, and the highest fidelity to Google's original weights, all without requiring any changes to your target model's configuration.