LLM routing is the secret sauce for cost-effective AI applications, but fine-tuning the router itself can yield surprising results. A recent experiment detailed on DEV.to demonstrates that scaling training data for a Qwen3-8B complexity scorer did not linearly improve accuracy. In fact, a model trained on just 50 synthetic examples achieved 86% accuracy, significantly outperforming a model trained on 2,500 examples, which settled at 76%.

The Experiment: Less Is More?

The project, known as Complexity Scorer, fine-tuned Qwen3-8B using the Tinker framework to assign a complexity score between 0.00 and 1.00 to user prompts. The goal was to route simple tasks, like JSON extraction, to cheaper models while reserving advanced reasoning models for complex system design questions. The baseline original Qwen3-8B model performed poorly, with only 50% accuracy on the evaluation set. However, the fine-tuned variants showed erratic performance gains relative to dataset size. While the 50-example model hit 86%, the 500-example model dropped to 74%, and the 2,500-example model barely recovered to 76%.

Why Data Quality Trumps Quantity

The author attributes this counterintuitive outcome to synthetic data quality and evaluation methodology. AI-generated training examples often contain inconsistent labels, where similar prompts receive different complexity scores, blurring the decision boundaries the model needs to learn. Furthermore, the evaluation set consisted of only 50 examples. This small sample size means the 86% result might reflect overfitting to that specific subset rather than true generalization. The marginal improvement from 74% to 76% when increasing data from 500 to 2,500 examples represents just one additional correct prediction, suggesting the gains are statistically negligible.

Technical Implementation and Open Weights

Beyond the training results, the project highlights the utility of open-weight models for infrastructure tasks. By using Qwen3-8B, developers avoid vendor lock-in and high API costs associated with using closed models for every classification request. The implementation includes a .NET Minimal API with rate limiting, API key authentication, and a browser-based playground. The model adapter was published to Hugging Face, allowing other developers to experiment with local deployment. This separation of concerns ensures that the complexity estimation step does not interfere with the final answer generation, optimizing both latency and cost.

Key Takeaways

  • Synthetic data inconsistencies can degrade model performance more than small dataset sizes.
  • Small evaluation sets (n=50) create high variance, making accuracy metrics unreliable for comparison.
  • Open-weight models like Qwen3-8B offer critical flexibility for building custom routing infrastructure.
  • Data quality and label consistency are more important factors for fine-tuning success than raw volume.

The Bottom Line

Don't blindly scale your training data; validate your evaluation set first. If your labels are noisy, more data just teaches the model to be confidently wrong.