The birthday reminder app Circles has demonstrated that specialized, small language models can outperform larger general-purpose models in specific on-device tasks. By fine-tuning a 270-million-parameter Gemma 3 model, the team achieved a parsing speed of approximately one second per dictated note on a Pixel 8 Pro. This approach allows the app to convert spoken family details into structured contact data without sending sensitive personal information to a server, maintaining both privacy and offline functionality.

The Failure of General-Purpose Small Models

Initial attempts to use Gemma 4 E2B with a long instruction prompt resulted in significant latency issues. While the model achieved 95% field accuracy on a test set, it required about 11 seconds on the GPU and 21 seconds on the CPU to process a single note. The bottleneck was not the prompt reading time, which took only 1.2 seconds, but the generation speed of roughly 10 tokens per second. Other off-the-shelf small models proved even less reliable, with some producing empty strings or broken JSON structures, making them unsuitable for real-time user interaction.

Fine-Tuning for Specificity and Speed

The solution lay in abandoning the general instruction-following approach in favor of direct fine-tuning on a synthetic dataset. The team trained the Gemma 3 270M model on 18,000 generated notes in English, Spanish, and Russian using MLX on a MacBook Air. The training process took only 40 minutes for 3,400 steps. By removing the complex prompt and instructing the model to output compact JSON directly, the inference time dropped to 1.0 second per note on the CPU. This efficiency was achieved while running the model via LiteRT-LM with int8 quantization, resulting in a manageable 296 MB file size. On a test set written after training, the tiny model was also more accurate than the model ten times its size with the long prompt.

Data Generation as the Core Engineering Challenge

The article emphasizes that the data generator, rather than the training code, was the critical component of success. The team iteratively refined the generator to mimic real-world dictation quirks, such as the absence of commas, mixed number formats, and self-corrections like 'the seventh โ€” no, the eighth.' Addressing linguistic nuances, such as Russian case inflections for names, significantly improved accuracy. For instance, teaching the model to handle multi-sentence notes increased full-note accuracy from 5 to 17 out of 24 in a specific test set. The final model correctly parses 14 out of 15 real dictated notes and 24 out of 24 'spoken' test notes.

Deployment and User Experience

To keep the initial app download small, the 296 MB model is delivered via on-demand packs: Play Asset Delivery for Android and Apple-hosted Background Assets for iOS 26.4 and later. This ensures users who do not use the dictation feature never download the model. The appโ€™s architecture separates extraction from logic; the model only extracts entities, while the app handles birth year calculations, duplicate merging, and grammatical relation formatting. This division of labor ensures that no data is saved until the user reviews the generated cards, a workflow where three out of four notes require no manual edits.

Key Takeaways

  • Specialized fine-tuning of small models (270M parameters) can outperform larger general models in speed and accuracy for narrow tasks.
  • On-device inference using CPU and int8 quantization can achieve sub-second latency, removing the need for flagship GPUs or NPUs.
  • Synthetic data generation that mimics real-world speech patterns is more critical than complex prompt engineering for constrained output formats.

The Bottom Line

The era of needing massive models for every task is over; Circles proves that a tiny, specifically trained model running on a standard CPU can deliver superior privacy and performance for structured data extraction. Small model fine-tuning is no longer just an optimization technique but a viable primary architecture for on-device AI applications. Investing in data generation pipelines yields higher returns than prompt engineering when dealing with constrained output formats like JSON.