ReOpenly published a benchmark this week challenging the assumption that frontier models are always the best choice for production tasks. The team fine-tuned Llama 3.1 8B on 60,000 legal contract clauses from the LEDGAR dataset for just $8.70 in GPU time. When tested against Claude Sonnet 5 on 437 held-out clauses, the small model achieved 73.7% accuracy compared to Sonnet's 70.0%, while answering 9x faster and costing 16x less per thousand requests.

The Economics of Specialization

The test highlighted a critical inefficiency in modern LLM deployment: paying for general knowledge when a narrow task requires only specific expertise. Claude Sonnet 5, running zero-shot with a 100-category prompt, took 2.23 seconds per answer and cost approximately $2.00 per thousand classifications. The fine-tuned Llama 3.1 8B responded in 0.25 seconds and cost $0.125 per thousand at ReOpenly's rate of $0.03 per GPU minute. This translates to a $22,500 annual saving for teams processing one million classifications per month.

Where Generalists Still Win

The data revealed a nuanced performance gap based on category frequency. On common categories, the fine-tuned model dominated with 83.6% accuracy, beating Sonnet's 72.0% by 11.6 points. However, Claude Sonnet 5 maintained a lead on rare categories, scoring 67.4% versus the fine-tuned model's 60.4%. This confirms that broad general knowledge remains valuable when training examples are scarce, though ReOpenly notes that better data collection for tail categories could close this gap.

Verification and Limitations

To ensure the results weren't specific to one model family, ReOpenly also fine-tuned Qwen3.5 9B, which achieved 72.1% overall accuracy. The benchmark used the public LEDGAR dataset, which contains long-tail distributions typical of real-world classification problems. While Meta previously reported even higher accuracy gains with private data, this public reproduction validates that small open models can match or exceed frontier performance on stable, high-volume tasks.

Key Takeaways

  • Fine-tuning Llama 3.1 8B cost $8.70 and paid for itself within 5,000 requests.
  • Small models offer 16x cost reduction and 9x speed improvements for narrow tasks.
  • Frontier models retain an advantage on rare categories with insufficient training data.
  • Qwen3.5 9B showed similar results, confirming the trend across model families.

The Bottom Line

If your AI task is stable, repeated at volume, and backed by real examples, paying for a frontier model is financial waste. The specialist beats the generalist.