The on-device AI landscape for Android has solidified into a rigid hierarchy in 2026, with LiteRT 2.3 serving as the default runtime and Gemini Nano acting as the platform LLM. The era of experimental edge inference is over; the current stack is production-ready, mature, and heavily optimized for privacy and latency. However, the critical insight for developers is to abandon flagship-centric assumptions. Success now depends on supporting the 'floor'—budget CPU-only devices—rather than just the NPU-equipped high-end segment.

The Runtime Wars: LiteRT vs. ONNX

TensorFlow Lite has officially rebranded to LiteRT, with version 2.3.0 released in October 2026. This runtime now features a CompiledModel API that automatically negotiates backends across Hexagon NPUs, Tensor TPUs, and Adreno/Mali GPUs. While ONNX Runtime Mobile remains a viable choice for PyTorch-first teams seeking binary parity with iOS, LiteRT holds the ecosystem advantage for Google-centric development. The raw speed difference between the two is negligible (~30-50ms on MobileNet), but LiteRT offers first-party integration with MediaPipe Tasks and the Gemini Nano path, making it the default for most Android apps.

Gemini Nano and the Platform LLM

Gemini Nano is no longer a downloadable model but a system service accessed via AICore on supported devices like the Pixel 11 and Galaxy S26. It offers ready-made GenAI tasks for summarization and proofreading, eliminating the need for prompt engineering or model downloads. For custom instructions, the Prompt API is in beta, while structured output remains in alpha. Crucially, Gemini Nano is restricted to flagship and foldable devices. Developers must implement robust fallback logic, degrading to bundled LiteRT-LM models (like Gemma INT4) or cloud Gemini when running on mid-range or budget hardware.

Quantization and Hardware Tiers

Model size and accuracy are managed through aggressive quantization. INT8 is the production standard for vision tasks, offering a ~75% size reduction with minimal accuracy loss (0.5-2%). For on-device LLMs, INT4 is necessary to fit models within memory constraints, despite a higher accuracy hit of 1-5%. The hardware landscape is stratified into four tiers: Flagship NPUs (Snapdragon 8 Elite Gen 5) handle 2-4B INT4 LLMs at interactive speeds, while Tensor TPUs (Pixel 11) run Gemini Nano natively. Mid-range GPUs and budget CPUs require graceful degradation, often routing heavier LLM tasks to the cloud while keeping latency-critical vision tasks local.

Key Takeaways

  • Plan for the Floor: Your app must function on budget CPU-only phones, not just NPU-equipped flagships.
  • LiteRT is Default: TensorFlow Lite is now LiteRT 2.3; use the CompiledModel API for automatic backend negotiation.
  • Gemini Nano is OS-Level: It runs on supported flagships via AICore; always implement fallbacks for unsupported devices.
  • Quantization Strategy: Use INT8 for vision models and INT4 for LLMs to balance size and accuracy.
  • NNAPI is Dead: Do not start new projects on NNAPI; migrate to LiteRT or ONNX Runtime Mobile.

The Bottom Line

On-device AI in 2026 is no longer about raw capability but about architectural discipline. The stack is mature, but the complexity lies in managing hardware fragmentation and fallback logic. If you aren't building for the budget tier, you aren't building for the Android market.