MLCommons has officially launched Candela, a benchmarking-as-a-service platform designed to quantify AI risk and reliability for both developers and enterprise deployers. The initiative, backed by a consortium of over 125 organizations including Microsoft, Google, NVIDIA, and OpenAI, aims to solve the persistent problem of inconsistent safety evaluations in large language models. By offering on-demand access to consensus-built AILuminate benchmarks, Candela seeks to provide a standardized metric for measuring how models handle hazards and resist adversarial attacks, moving beyond the fragmented landscape of open-source leaderboards.

The Problem with Open Benchmarks and LLM Judges

A core pillar of Candela’s credibility is its decision to keep test prompts strictly closed, accessible only to MLCommons staff. This approach directly addresses benchmark contamination, where models are inadvertently fine-tuned on public test sets, inflating their apparent safety scores. Furthermore, the platform rejects the common industry practice of using LLM-as-a-judge for scoring. Instead, MLCommons employs a private, custom-built machine learning classifier trained specifically for safety evaluation. This classifier provides stable, comparable results over time, avoiding the stochastic variability inherent in generative grading models, which often fail to reliably detect when a system has fallen for a jailbreak technique.

New Safety and Jailbreak Standards

Candela debuts with updated versions of the AILuminate benchmarks: Safety 1.2.1 and Jailbreak 1.1. Safety 1.2.1 evaluates systems across twelve hazard categories, ranging from violent crime to specialized financial and legal advice. The Jailbreak 1.1 benchmark tests eleven of these categories using a taxonomy of adversarial techniques, including simple typos and lookalike characters, as well as complex context framing and deception. Initial results for Safety 1.2.1 are already public for approximately 30 open-weight models, with Jailbreak data slated for release soon. The methodology relies on item response theory, the same statistical framework used in standardized tests like the SAT, to estimate prompt difficulty and separate stronger systems from weaker ones.

Enterprise Adoption and Governance Implications

The platform distinguishes itself by allowing tests on the full deployment stack, not just the base model. Through a local client relay, organizations can evaluate how system prompts, retrieval-augmented generation (RAG), and agent loops alter safety behavior. This is critical because safety scoring is not composable; a model that passes safety checks in isolation may fail once wrapped in a complex enterprise harness. Candela offers three tiers of results: inexpensive practice tests for internal development, annotated results for detailed debugging, and official results suitable for public citation in system cards and procurement decisions. Major players like Microsoft and NVIDIA have already endorsed the platform, with Google DeepMind demonstrating a confidential evaluation process that protects both model weights and test prompts.

Key Takeaways

  • Candela uses closed prompts and a custom ML classifier to prevent benchmark contamination and ensure stable scoring.
  • The platform tests the entire AI system stack, acknowledging that safety behavior changes with RAG and agent configurations.
  • Results are standardized using item response theory, allowing for consistent comparison across different models and time periods.
  • Official results are designed to be citable for governance, procurement, and regulatory compliance.

The Bottom Line

Candela represents a maturation of the AI safety market, shifting from academic curiosity to industrial-grade compliance tooling. By rejecting LLM-as-a-judge in favor of deterministic classifiers and closed datasets, MLCommons is attempting to build the trust infrastructure necessary for enterprise AI adoption. If successful, this could become the de facto standard for AI risk assurance, effectively turning safety from a vague promise into a measurable, auditable metric.