Shalendra Gaurβs latest benchmark on DEV.to exposes the fragility of applying Large Language Model outputs directly to stateful systems. Using Qwen2.5-7B-Instruct, a 4-bit quantized model running on a Colab T4 with greedy decoding, the test revealed that raw model replies corrupted the internal state in 12 out of 12 episodes. The model consistently failed to cascade deletions, leaving orphaned connections in the service graph despite the logical necessity of the rule.
The Failure of Raw Application
The experiment involved 12 episodes of 30 sequential requests, each editing a service graph. The critical ruleβthat deleting a service must also delete its connectionsβwas deliberately omitted from the prompt. When the modelβs proposals were applied as written, the state was exactly correct in only 30 of 360 steps, an 8% success rate. None of the 95 proposals that attempted to delete a service were complete; every single one left behind dangling edges, proving that the 7B model lacks the inherent logic to handle implicit cascade constraints without explicit instruction.
The Gate and Repair Solution
The test evaluated the AXI architecture, a host-side layer by Rajnish Choubey that intercepts and modifies model outputs. A simple gate, which only refuses bad proposals without fixing them, prevented corruption (0 corrupted steps) but resulted in a state that was 23.3 items away from the correct target. In contrast, the Gate and Repair mechanism, which actively completed the refused proposals, achieved 99% accuracy. The state was exactly right in 358 of 360 steps, and 11 of the 12 episodes were perfect at the final step, demonstrating that host-side logic can compensate for small model deficiencies.
Scaling and Limitations
While the repair layer showed dramatic improvements at the 7B scale, the gap narrows with larger models. On a 14B model, applying outputs as written still corrupted 24 of 140 graphs when the rule was unstated, though the gate and repair mechanisms prevented any corruption in 840 single requests. However, the system struggles with ambiguous requests that do not name a specific service, such as "remove every edge with weight w." In these cases, the scope filter had nothing to read, leading to errors in 11 of 280 outputs at both model sizes. This highlights that while host-side repair is powerful, it relies on the model providing at least a partial, identifiable signal to act upon.
Key Takeaways
- Qwen2.5-7B-Instruct failed to handle implicit cascade deletions in every test episode when applied raw.
- A host-side repair layer (AXI) increased accuracy from 8% to 99% by completing incomplete model proposals.
- The repair mechanism is most effective on smaller models; the benefit diminishes but remains present at 14B.
- Ambiguous requests lacking specific service names remain a failure point for the scope filter regardless of model size.
The Bottom Line
Small models are prone to silent logical failures in stateful applications, but host-side repair layers offer a robust safety net that significantly outperforms raw application or simple gating.