Microsoft is finally meeting developers where they live: in the open-source ecosystem. Today, Windows ML introduced experimental support for llama.cpp, allowing developers to run GGUF models locally through new task-specific APIs. This move acknowledges that while ONNX is the enterprise standard, llama.cpp is the de facto tool for rapid experimentation with the latest open-weight models. The update is particularly relevant for users on new hardware like the Surface Laptop Ultra powered by NVIDIA RTX Spark.

Bridging the Inference Gap

The core of this release is the integration of the GGUF community's preferred runtime into the Windows ML stack. By pulling a model from Hugging Face and running it via the new Text Generation API, developers can bypass the friction of format conversion. The API exposes an OpenAI-compatible endpoint, meaning you can prototype against local models using the same SDK you already know. This lowers the barrier to entry significantly, letting you point base_url to localhost and start streaming tokens without rewriting your application logic.

Native Runtime and Arm64 Expansion

Beyond the high-level APIs, Microsoft is shipping an experimental Windows-native Runtime API. This layer offers zero-copy paths for images and audio, deterministic multi-model pipelines, and ahead-of-time compilation. It runs side-by-side with the familiar ONNX Runtime, allowing teams to adopt native optimizations incrementally. Simultaneously, PyTorch now offers official native Windows Arm64 CPU builds, with NVIDIA publishing CUDA-enabled packages for supported hardware. This extends the full model lifecycle—training, fine-tuning, and inference—to Windows on Arm devices.

Key Takeaways

  • Windows ML now supports GGUF models via experimental llama.cpp integration, accessible through OpenAI-compatible endpoints.
  • A new experimental Windows-native Runtime API enables zero-copy data paths and deterministic multi-model pipelines.
  • PyTorch and Triton now have official native support for Windows Arm64, closing the gap with x64 workflows.
  • The Windows ML CLI allows for analyzing, optimizing, and benchmarking models before deployment.

The Bottom Line

Microsoft is clearly trying to stop fighting the open-source tide and start riding it. By making llama.cpp a first-class citizen in Windows ML, they’re signaling that local, on-device AI is no longer just for cloud giants—it’s for the guy building a chatbot on his laptop.