A team of researchers from Stanford and Semifinal has published what might become a foundational framework for evaluating local AI inference: intelligence per watt (IPW), which measures task accuracy per unit of power consumed. The study, posted to arXiv in November 2025 with revisions through August 2026, evaluated over 20 state-of-the-art small language models (under 20 billion active parameters) across eight hardware accelerators using one million real-world single-turn chat and reasoning queries.

Defining the New Efficiency Standard

The researchers argue that existing benchmarks fail to capture what matters for local deployment decisions. Raw accuracy metrics ignore power constraints, while simple throughput numbers don't account for quality of outputs. IPW solves this by combining accuracy (measured as local model win rate against frontier models) with energy consumption into a single comparable metric. The authors tested configurations including Apple's M4 Max as a representative local accelerator alongside cloud hardware running identical model weights.

What the Numbers Show

Local language models successfully answered 88.7% of real-world queries without requiring frontier model escalation—a finding that immediately challenges assumptions about needing cloud-scale infrastructure for production workloads. Longitudinal analysis spanning 2023 to 2025 reveals IPW improved by 5.3x, driven by both algorithmic advances in smaller models and hardware improvements in local accelerators. Perhaps most striking: the percentage of queries that could be handled locally rose from just 23.2% in early data to 71.3% by 2025's latest benchmarks.

Hardware Efficiency Gap

The study's clearest finding for infrastructure teams: local accelerators achieved at least 1.4x lower IPW (meaning better efficiency) than cloud hardware running the same models. This suggests meaningful optimization headroom remains in local inference stacks, though it also demonstrates that on-device processing can already compete with centralized alternatives for a significant query subset. Latency and power characteristics varied considerably by domain, with reasoning-heavy tasks presenting different tradeoffs than simple chat interactions.

Key Takeaways

  • Local inference now handles 71% of queries effectively, up from 23% in just two years
  • Small models under 20B parameters can match frontier model quality on most everyday tasks
  • Apple M4 Max and similar local hardware outperform cloud alternatives on efficiency by at least 40%

Developer Implications

For developers building AI-powered applications today, the IPW framework provides concrete guidance for architecture decisions. The research suggests that blanket reliance on API calls to frontier models may be economically and environmentally inefficient for a majority of use cases. Running smaller models locally isn't just feasible—it's demonstrably more efficient when power consumption matters. The 88.7% success rate means most queries never need cloud escalation if developers invest in appropriate model selection.

The Bottom Line

This research should make every developer reconsider their default assumption that AI inference requires remote API calls. Local hardware efficiency has crossed a threshold where the tradeoff analysis favors on-device processing for most production workloads, not just edge cases or toy examples. The IPW metric itself is the real contribution here—giving the industry a standardized way to compare these tradeoffs rather than debating benchmark methodology in circles.