Ahmet Koc, the developer behind ultraielts, has implemented a strict quality assurance pipeline that prevents any new IELTS practice question from going live until it survives a 'blind solver' test. The free, Turkish-interface site offers computer-delivered mock tests in English, and Koc recognized that the most damaging bug in exam prep isn't a crash, but a question with two defensible answers or a wrong key. To mitigate this, he built a system where an AI model attempts to answer questions with the exact same limited context a human candidate sees, ensuring the test's logic holds up under scrutiny before a student ever clicks 'submit'.

The Blind Solver and Production Scoring

The core of the validation process is the blind solver, which receives only the passage or transcript, the questions without answers, and any relevant images. It is instructed to answer as a Band 9 candidate, providing confidence levels and reasoning. Crucially, the solver's output is not checked via simple string equality. Instead, it is scored by the site's actual scoreGroup function, the same code that grades real users. This approach eliminates false positives caused by acceptable variations, such as 'colour' vs 'color' or '7:30' vs '7.30', ensuring that if the checker flags a mismatch, it represents a genuine ambiguity or error in the question design.

Disagreement Analysis and Automated Repairs

The pipeline runs the solver twice at different reasoning-effort settings to identify three types of mismatches: strong, weak, and consensus. A 'consensus' mismatch, where both AI runs agree on a wrong answer, often indicates a missing acceptable variant in the key. These findings are passed to a secondary 'reviewer' pass, which analyzes the context to determine if the key is wrong, ambiguous, or correct. If an error is confirmed, an automated repair step attempts to fix the question group using precise find-and-replace edits. However, these repairs are heavily constrained; they are rejected if they change question numbers, fail to apply cleanly, or increase the total validator error count.

The Publish Gate and Rights Management

No test is published without passing a final gate that refuses to go live if the validator finds errors or if the QA job has open, unwaived issues. Beyond technical correctness, Koc places heavy emphasis on intellectual property rights. Any content imported from third-party books remains private, accessible only via share links for specific students. Only original or rights-cleared tests are allowed to become public, protecting the site from the common pitfall of using 'real past papers' with dubious origins. This dual focus on logical integrity and legal clearance ensures the platform remains a trustworthy resource for exam preparation.

Key Takeaways

  • Mirror the user experience: The AI checker must see exactly what the student sees, using the same data stripping functions.
  • Use production code for validation: Scoring AI outputs with the same logic as human inputs prevents false alarms from acceptable variants.
  • Constrain automated fixes: Repair steps should be strictly limited to prevent introducing new errors or altering question structure.
  • Separate detection from judgment: Use one pass to find disagreements and a second, context-aware pass to determine the root cause.

The Bottom Line

Koc's pipeline is a masterclass in defensive coding for content platforms; by treating AI solvers as adversarial users and enforcing production-grade scoring, he built a QA system that protects user trust better than any manual review could.