A new independent evaluation, dubbed HinglishBench, has exposed a significant efficiency penalty for the Beam-1.0-501B-A23B model when processing romanized Hindi-English code-mix, or Hinglish. While accuracy remains largely stable across languages for most tested models, Beam requires 1.47x more completion tokens to solve identical math problems in Hinglish compared to English. This finding challenges the assumption that multilingual capabilities in modern LLMs are cost-neutral, specifically for models like Beam which, according to its model card, was developed predominantly for English.

The Token Tax Reality

The benchmark utilized 82 GSM8K math problems, translated into Hinglish and verified by automated numeric-preservation rules. Three models were tested: Beam-1.0-501B-A23B, Gemma-3-27B-it, and Qwen3-32B. The results were distinct. Beam maintained high accuracy (95.1% in Hinglish vs. 96.3% in English) but saw its median token ratio spike to 1.47x. In contrast, Gemma showed no token increase (0.96x ratio) and actually improved slightly in accuracy, while Qwen3-32B suffered an accuracy drop to 82.9% but maintained a neutral token ratio of 1.00x.

Model-Specific Vulnerabilities

This is not a universal law of Hinglish processing; it is a model-specific artifact. The data suggests that Beamโ€™s internal representation or decoding strategy struggles with the romanized script, forcing it to generate more verbose reasoning chains to reach the same conclusion. The study notes that this is a token tax rather than a monetary bill, but given that serving costs scale linearly with output tokens, this efficiency gap translates directly to infrastructure costs for developers deploying Beam in Indian markets.

Limitations and Methodology

The author, Sansk Ya, disclosed that the benchmark was built and run by an AI assistant, with human direction. The dataset is limited to a single run per cell with n=82, meaning these are observed values, not statistically proven effects. Furthermore, the Hinglish translations were generated by Beam itself and checked by automated rules, not human linguists. This self-referential translation method introduces potential artifacts, though the pattern held in additional reverse-direction tests where Hinglish was the source language.

Key Takeaways

  • Beam-1.0 incurs a 1.47x token cost for Hinglish queries compared to English, despite similar accuracy.
  • Gemma-3-27B-it and Qwen3-32B did not exhibit a token tax, though Qwen saw a 7.3 percentage point accuracy drop.
  • The study highlights the importance of language-specific efficiency benchmarks beyond standard accuracy metrics.
  • Results are preliminary, based on single runs and automated validation, lacking human linguistic review.

The Bottom Line

If you are building for the Indian market, do not assume English-only benchmarks predict your Hinglish costs. Beamโ€™s token bloat is a silent killer for inference budgets, while Gemma proves that code-mix can be handled efficiently with the right architecture. Test your specific language pairs before scaling.