For three years, on-device AI was mostly marketing theater. Demos ran rough, quality lagged cloud by a generation, and every serious feature still resolved to a per-token bill in someone's datacenter. That story is dead as of mid-2026. Apple's third-generation Foundation Models dropped at WWDC on June 8, and Google's Gemma 4 family arrived April 2—two releases that quietly moved the floor. Genuinely useful agents now run on hardware you already own, offline, for free.

The Economics Nobody Priced In

Here's the load-bearing fact nobody talks about in benchmark threads: when your AI lives in the cloud, every inference is metered. Input tokens, output tokens, a line item that scales linearly with usage—and explodes the moment you wrap a model in an agent loop. A single "go do this task" can fan out into dozens of calls as the agent plans, calls tools, retries, and re-reads its own output. The bill grows with your ambition. Move that same workload to device and marginal cost is approximately $0. No API key, no rate limit, no usage dashboard. You paid for the silicon once; every token after that is free in the only sense a product manager cares about.

Sparse Beats Big: The Architecture That Did It

Apple's AFM 3 on-device model carries roughly 20 billion parameters but fires just one to four billion per request. That's not a limitation—it's the entire thesis. Apple's Instruction-Following Pruning keeps the full model in flash and swaps only the relevant "experts" into DRAM as needed. The phone never holds 20B of active weights; it streams the slice required for each token. Google's Gemma 4 attacks the same problem with Per-Layer Embeddings: the E4B edge model carries roughly 8B total parameters but runs with about 4.5B effective. Its bigger sibling, a 26B mixture-of-experts model, only lights up a fraction of experts per token. MoE and IFP are the same insight wearing different clothes—most of any large model is dead weight on any single token, so don't pay to run it.

These Are Not Toy Models Anymore

The capability jump is real, and it's broadest where everyday use matters: multimodality. AFM 3's on-device model takes images in, and Apple reports human raters preferred its image understanding roughly 61% of the time over the previous generation. Its text-to-speech scored 4.24 on a 5-point mean-opinion scale versus 3.82 for baseline—roughly the gap between "obviously a robot" and "fine, I'll actually listen to this." Gemma 4 ships native vision and audio, 128K context on edge models, and support for over 140 languages. Google's own framing—that these models "outcompete models 20x their size"—is the whole thesis in one line.

What It Still Can't Do

The honest caveat: device models are not frontier models, and pretending otherwise is how you ship a disappointing feature. Hard multi-step reasoning, long-horizon coding, deep research across large corpora—those still belong in the cloud with much larger models and big context budgets. MMLU benchmarks that get thrown around for 14B-class models? That test is saturated and gameable; a leaderboard score tells you almost nothing about whether something can hold a five-step plan together. The right mental model is hybrid: the device handles fast, private, high-frequency work and hands off to cloud only when a task genuinely outgrows it. The interesting engineering of 2026 isn't the models—it's the routing layer that decides which is which.

Apple Opened the Gates

The most underrated WWDC announcement wasn't the model itself—it was the door. Apple opened its Foundation Models framework to third-party and open models, with Swift packages for Anthropic's and Google's models on the way, plus agentic primitives and on-device semantic search added to the SDK. Translation: developers can write apps against one local-first AI framework and let the device decide which model answers. That's the platform move. The model becomes a commodity inside it; the framework—agent primitives, the semantic index over your files, the routing logic—is the actual moat. Once the OS ships a free, private, capable model with a clean API, "add AI" stops meaning "add a cloud dependency and a billing relationship" and starts meaning "call a system function."

The Take

The cloud-AI era trained everyone to assume intelligence is a utility you rent by the token. 2026 is the year that assumption cracked at the edge—not because device models got as smart as the frontier (they didn't), but because sparse architectures finally made "large but cheap to run" a real category, and the economics of zero marginal inference are too good to ignore for the enormous class of features that never needed a genius in the first place. The cloud keeps the hardest problems. The device quietly takes everything else—offline, private, off the meter. Most software hasn't been rewritten to assume this yet. The teams that rewrite first will look, briefly, like magicians.