Accuracy is a lie we tell ourselves when evaluating LLMs. JGENai and VIDRAFT have dropped a new metacognition benchmark that stops asking if the model got it right and starts asking if the model knows it might be wrong. The release includes a 400-question adversarial dataset, a 24-model public leaderboard, and 11 model-specific adapters designed to probe self-error detection without touching base weights.

The Trap Question Strategy

The core of the benchmark is 400 "trap" questions where the superficially plausible answer is incorrect. This design forces the model to pause and re-examine its reasoning rather than pattern-matching to a confident-sounding output. Evaluation is split into two distinct axes: standard multiple-choice accuracy and open-ended self-assessment. The latter tests whether the model can correctly judge if its own free-form answer is likely wrong after generation.

Adapters as Non-Invasive Probes

The 11 released adapters are lightweight add-ons that attach to existing models without modifying architecture or trained weights. They analyze internal signals produced during answer generation to compute a risk score, indicating the likelihood of incorrectness. This output is a separate signal that downstream systems or human reviewers can consume independently. For developers, this means wrapping a deployed model with a metacognition layer requires no retraining or fine-tuning, offering a drop-in reliability probe for high-stakes applications.

Leaderboard Results and the Accuracy Gap

JGOS-31B-Citizen, developed by JGENai CEO Sunyoung Choi, ranked #1 out of 24 models with a 99.5% trap-avoidance rate, answering 398 of 400 questions correctly. It is also the top-ranked model on the K-AI Leaderboard operated by Korea's Ministry of Science and ICT (MSIT) and the National Information Society Agency (NIA). However, the results highlight a critical disconnect: several models performed strongly on multiple-choice accuracy yet showed limited capability in open-ended self-assessment. This empirically confirms that raw accuracy and metacognitive awareness are distinct competencies.

Key Takeaways

  • The benchmark focuses on self-error detection, arguing that a model not knowing it is wrong is more dangerous than being wrong.
  • Adapters provide a risk signal without retraining, making them viable for production environments in medicine, law, and finance.
  • JGOS-31B-Citizen leads the 24-model leaderboard, but the gap between accuracy and self-awareness remains significant across the board.
  • Artifacts are available on Hugging Face under the JGENai organization, though exact repository slugs for adapters require manual navigation.

The Bottom Line

We have spent years optimizing for accuracy while ignoring the silent failure mode of confident incorrectness. By decoupling self-error detection from raw performance, JGENai and VIDRAFT have finally given developers the tool to identify which models are actually safe for high-stakes deployment.