The debate between brittle heuristic routing and intelligent model dispatching just got a concrete data point. A new experiment detailed on DEV.to tested whether a decision model could replace hand-written logic in a fictional banking backend, Kestrel Bank. Using the Banking77 test set of 3,080 messages, the study compared four dispatchers: traditional rules, TypeSafe's jev-1.13.0, OpenAI's gpt-6-luna, and a hybrid approach. The results suggest that while LLMs offer competitive accuracy, specialized decision models provide a compelling speed advantage that rules-based systems simply cannot match.

Accuracy and Latency Tradeoffs

The experiment yielded nuanced results that defy simple 'model vs. rules' narratives. The rules-based dispatcher achieved 65.5% exact-action accuracy but required 772 lines of implementation code. In contrast, JEV hit 80.9% accuracy with only 60 lines of code, while the LLM (gpt-6-luna) edged slightly ahead at 81.5% with 95 lines. However, the latency gap was stark: JEV processed decisions in a median 111 ms, nearly nine times faster than the LLM's 975 ms. The rules engine was fastest at 0.29 ms, but its accuracy ceiling was significantly lower, abstaining on 17% of inputs.

The Cost Paradox and Hybrid Strategy

Surprisingly, the LLM proved cheaper per 1,000 requests ($0.028) compared to JEV ($0.083), largely due to cached-input pricing benefits for repeated action catalogues. The hybrid approach, which accepted agreements between rules and JEV, achieved the highest accuracy on answered messages (90.7%) but referred 18.3% of traffic to humans. This strategy reduced incorrect automatic selections compared to single-model routers, though it retained the complexity of both underlying systems. The study notes that this comparison is exploratory and does not establish superiority on future traffic, particularly given the hybrid's reliance on a fixed threshold of 0.9 confidence for disagreement resolution.

Safety Beyond Selection

A critical distinction emerged between selecting the right action and safely executing it. Of the hybrid's 233 incorrect selections, 32 chose handlers capable of changing bank state, such as blocking a card or opening a claim. The authors emphasize that correct selection does not equate to safe execution, especially since the evaluation logs recorded predictions rather than real customer outcomes. This highlights a gap in current model-based routing: without additional confirmation layers or state-aware checks, high-confidence model decisions can still lead to harmful automated actions.

Key Takeaways

  • JEV offered a 9x speed advantage over LLMs with comparable accuracy, making it ideal for latency-sensitive routing.
  • Hand-written rules reached a performance ceiling around 65% accuracy, requiring extensive maintenance code.
  • Hybrid strategies improved precision but increased system complexity and human referral rates.
  • Model selection accuracy does not guarantee execution safety, particularly for state-changing banking actions.

The Bottom Line

While the hybrid approach offered the best precision, the JEV model's balance of speed, accuracy, and low code footprint makes it the superior choice for most production routing tasks, provided developers implement strict safety checks for state-changing actions.