In a demonstration of practical LLM-assisted optimization, the open-source project fast-melroformer has successfully tuned the Mel-Band RoFormer voice separation model to run at 4.19x real-time speed on an 8 GB M3 MacBook Air. The project, which credits Claude for assisting in the optimization process, leverages Apple Silicon’s Neural Engine (ANE) and custom Metal kernels to drastically reduce inference time. By processing a 45-minute TV episode in roughly 11 minutes instead of the previous 42, the project proves that specialized hardware acceleration can yield massive gains even for complex transformer architectures without sacrificing audio fidelity.

The Optimization Pipeline

The performance leap was achieved through a step-by-step architectural overhaul, documented in the project’s NOTES.md. Starting with a stock PyTorch fp32 model running on the GPU (MPS) at 1.08x real-time, the developer first applied fp16 quantization, bringing speeds to 1.35x. The most significant early gain came from fixing rotary embeddings (RoPE) and RMSNorm, which were previously running in fp32 and rebuilding angle tables on every call. This inefficiency consumed 30 to 46% of attention time on MPS; patching this alone pushed performance to 2.19x. Further gains were made by folding output gates into the QKV matrix multiplication and implementing custom Metal kernels for attention glue, reaching 2.62x.

Neural Engine Integration

The final breakthrough involved moving the 12 transformer blocks to the Neural Engine via Core ML, which pushed speeds to 3.61x. To accommodate the ANE’s memory constraints, time attention was split into small tiles, achieving the final 4.19x speed. Interestingly, the developer noted that quantization did not help speed because 88% of the weights reside in mask estimators that account for only 4% of the runtime. The project uses an unmodified public checkpoint (vocals_mel_band_roformer.ckpt, 228M parameters), ensuring that the gains are purely architectural and hardware-specific rather than model-retraining dependent.

Performance and Limitations

The optimized model matches the fp32 reference output at 56.5 dB SDR over a 3-minute clip, a difference described as inaudible. However, the project warns that quiet passages may drop to 40 dB SDR, which is near the threshold of perceptible difference. The implementation requires macOS 15 and uv, with a one-time 5-minute compilation step for the Neural Engine model. Users should note that running two instances simultaneously on an 8 GB machine can cause freezes, and thermal throttling on fanless devices like the M3 MacBook Air may reduce speeds slightly below 4x during long files.

Key Takeaways

  • RoPE Inefficiency: Rotary embeddings running in fp32 with repeated table rebuilds were a major bottleneck, costing up to 46% of attention time on MPS.
  • ANE Offloading: Moving transformer blocks to the Neural Engine provided the largest single speed boost, jumping from 2.62x to 3.61x.
  • Quantization Limits: Weight quantization offered no speed benefit because the majority of weights are in mask estimators, not the time-critical transformer blocks.
  • Hardware Specificity: The 4.19x speedup is measured on an 8 GB M3 MacBook Air; thermal constraints on fanless devices may impact sustained performance.

The Bottom Line

While Claude provided the scaffolding, the real story here is the brutal efficiency of Apple’s Neural Engine when properly leveraged by a human expert willing to rewrite low-level kernels. This project serves as a stark reminder that generic LLM optimizations often fail without deep, hardware-specific architectural insights, but when they align, the results are transformative for local AI workloads.