A new open-source study from Munim, Inc. has delivered a clean null result that challenges a persistent assumption in AI-assisted education: that small language models silently "correct" student misspellings when quoting them. The research, designated as study v5, tested whether the presence of a misspelled word in a student’s essay reduces the fidelity of an AI-generated quote. The answer, for the two 3-billion parameter models tested, is no.

The Preregistered Test

The study design was fixed before any data generation, addressing the underpowered nature of its predecessor (v4), which failed to meet the preregistered floor of 150 misspelled-passage quotes. For v5, the team collected a fresh sample of 120 English-learner essays from the ELLIPSE corpus, split evenly between low and high writing-conventions bands. They deployed two popular small open-weights models, llama3.2:3b and qwen2.5:3b, via a local Ollama instance with prompt caching disabled to ensure raw model behavior. The models generated feedback in both English and Spanish, resulting in 960 total requests.

Statistical Verdict: No Clear Difference

The powered test successfully cleared its data floor, capturing 403 misspelled-passage quotes and 1,013 clean quotes. The pooled exact-quote rate was 91% for passages containing misspellings (366 of 403) versus 94% for clean passages (948 of 1,013). The resulting gap of -2.8 percentage points yielded a 95% bootstrap confidence interval of [-6.6, +1.1]. Because this interval spans zero, the preregistered rule dictates a verdict of "no clear difference." The suggestive penalty observed in the earlier underpowered study did not survive the increased sample size.

Models Respect Student Errors

Beyond the null hypothesis, the study revealed a reassuring descriptive finding: when models did quote a passage with a misspelled word, they retained the original spelling exactly 97% of the time (836 of 862 instances). This suggests that these small models are not aggressively normalizing text inside their quotes, a critical feature for teachers who need to verify that AI feedback corresponds to the actual student work rather than a sanitized version of it.

Key Takeaways

  • The study is the first powered, preregistered measurement of quote fidelity relative to source misspellings.
  • Results apply specifically to 3B-class models (llama3.2 and qwen2.5) running locally.
  • The code, data IDs, and analysis are fully open-source under Apache-2.0 and CC BY 4.0 licenses.
  • Two free tools, quote-check and quote_batch.py, were released to help educators verify quote accuracy independently.

The Bottom Line

While this negative result might seem minor, it is a vital calibration for the AI education tooling ecosystem. It proves that small, efficient models can be trusted to quote student work verbatim, even when that work is messy, without requiring the computational overhead of larger models to preserve textual integrity.