Cognition’s FrontierCode benchmark, now in version 1.1, challenges the assumption that modern LLMs are ready to autonomously maintain open source projects. Unlike standard coding evaluations that prioritize test passage, FrontierCode assesses whether AI-generated changes meet the stringent quality bars of actual repository maintainers. The benchmark was developed with input from maintainers of 36 open source repositories, who dedicated over 40 hours to crafting each task. This human-in-the-loop design ensures the metric reflects real-world friction, not just syntactic correctness.

The Maintainer Standard

The core innovation of FrontierCode is its holistic grading system. It penalizes models for behaviors that annoy maintainers, such as creating excessive diffs, modifying files outside the requested scope, or adding unhelpful tests. The verifier combines deterministic hidden tests with qualitative LLM reviews against maintainer-defined criteria. An attempt scores zero if it fails any blocking check, ensuring that only changes meeting all essential requirements receive a score between 70% and 100%. This approach highlights that passing tests is necessary but insufficient for production-ready code.

Performance Metrics and Model Rankings

Data from the 100-task FrontierCode 1.1 Main set reveals that even the top models struggle. Claude Opus 5.5 (max with fallback) leads the leaderboard with an average score of 52% and a pass rate of 58%. Following closely are GPT-6.1 Sol (max) and Claude Fable 5.1 (max with fallback), both scoring 49%. Notably, nine models from six different labs cluster within a tight 43% to 52% range, indicating a ceiling in current autonomous coding capabilities. The open-weights leader, MiMo-V2.6-Pro, achieves 45%, demonstrating that smaller, specialized models are becoming competitive with proprietary giants.

Cost Efficiency and Token Usage

Cost analysis shows significant disparities in efficiency. GPT-6.1 Sol (max) offers the best value, scoring 49% at just $0.81 per task, whereas Claude Opus 5.5 costs $10.15 per task for a marginal 3-point gain. Token usage further differentiates the models: GPT-6 variants use approximately 27k-33k output tokens per task, while Claude Sonnet 5.5 consumes nearly 236k tokens. This inefficiency translates to longer processing times, with some tasks taking up to 46 minutes to complete, compared to just 3 minutes for the fastest models like Inkling (xhigh).

Key Takeaways

  • Leading models fail more than 40% of tasks when judged by maintainer standards.
  • GPT-6.1 Sol (max) provides the best cost-to-performance ratio at $0.81 per task.
  • Nine models from six labs score within a narrow 9-point band, showing converged capabilities.
  • Claude Opus 5.5 is the most expensive per-task performer among top-tier models.

The Bottom Line

AI agents are not yet ready to replace human maintainers; they are expensive, inefficient assistants that still require significant cleanup before their code is mergeable. FrontierCode exposes the gap between 'working code' and 'maintainable code,' a distinction critical for enterprise adoption. Until models can consistently respect scope and minimize diff noise, human review remains non-negotiable.