Tiny AI Maker, the browser-based tool for training small text classifiers, has updated its reporting mechanism to include 5-fold cross-validation. Until this week, the tool relied on a single 80/20 split, a metric that often provides a misleadingly optimistic view of model performance on tiny datasets. The update addresses a critical pain point for developers working with limited data: the variance inherent in small test sets.

The Cost of Single-Split Illusions

The disparity between single-split and cross-validated scores is stark in the tool's AG News sample. This 200-row dataset, balanced with 40 rows per class, initially reported a 70.0% accuracy on a single holdout split. However, running a 5-fold cross-validation on the same data revealed a mean accuracy of just 57.0%, with a standard deviation of 5.7 percentage points across folds. The range spanned from 47.5% to 62.5%, highlighting that the original 70% figure was likely a statistical outlier rather than a reliable performance metric.

Calibration Metrics Reveal Softmax Overconfidence

Beyond accuracy, the Maker now calculates Brier scores and Expected Calibration Error (ECE) to assess how well the model's predicted probabilities align with actual outcomes. For the AG News sample, the model demonstrated poor calibration with a Brier score of 0.523 and an ECE of 0.139, indicating that the softmax outputs were not true probabilities. Even on the easier SMS spam task, which maintained consistent 90% accuracy across both methods, the model showed slight overconfidence, with 79 predictions landing in the highest confidence bin while only achieving 96.2% accuracy in that range.

Temperature Scaling Offers Partial Fixes

The update introduces nested temperature scaling to mitigate calibration errors. By fitting a single temperature parameter out-of-fold, the tool reduced the ECE for the AG News model from 0.139 to 0.074 and for the SMS model from 0.066 to 0.032. While this improves the reliability of the probability estimates displayed in the UI, it does not fundamentally rescue the four-class model's poor calibration, as the Brier score remains high at 0.503. Importantly, these scaling factors are not applied to the exported int8 weights, meaning the deployed models remain uncalibrated by default.

Key Takeaways

  • Single 80/20 splits on 200-row datasets can overstate accuracy by up to 13 percentage points compared to 5-fold CV.
  • New calibration metrics (Brier score, ECE) reveal that softmax outputs are often overconfident, even on high-accuracy tasks.
  • Temperature scaling improves probability reliability in the UI but is not exported to the final model weights.
  • The exported model remains the 80/20 fit, while the model card now explicitly documents the CV gap and calibration stats.

The Bottom Line

Reporting the worse number is the honest choice. A 200-row classifier that says 70% on one split and 57% on five folds is still useful as a demo, but it is not useful as a silent 70%.