🔍 Read the full analysis: FrontierCode: Can AI-Generated Changes Earn A Maintainer’s Approval? on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis evaluated 23 AI models on FrontierCode, a coding benchmark that tests whether a change meets repository-specific checks and maintainer criteria. Claude Opus 5.5 scored highest at 52.2%, while 57.7% of its attempts passed every blocking check; results show substantial gaps remain, and performance comes at widely differing costs.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark designed to assess whether maintainers would accept an AI-generated change, and Claude Opus 5.5 scored highest at 52.2%. The leading model passed every blocking check in 57.7% of attempts, leaving a substantial share that did not clear the benchmark’s required checks.
FrontierCode’s score is not a simple count of green tests. The benchmark evaluates patches using hidden tests, command checks, and reverse tests, which assess whether an agent’s tests fail on the original code. It also uses adaptive tests for complex behavior, limits on which files and how much code can be changed, and an LLM review against maintainer criteria. A failed blocking check makes an attempt score zero; patches that pass all blocking checks receive a score from 70% to 100%, depending on quality checks.
In Artificial Analysis’s results, GPT-6.1 Sol scored 49.1% and Claude Fable 5.1 scored 49.0%, placing them second and third. Claude Sonnet 5.5 scored 45.8%. Among open-weight models, Xiaomi’s MiMo-V2.6-Pro led at 45.4%, followed by DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%. Nine of the benchmark’s 100 Main-set tasks were not solved by any model on any attempt, according to the source report.
Artificial Analysis used the same open-source agent harness, Stirrup, for every model, allowing up to 250 turns per attempt and three attempts per task. The review stage used GPT-6 Luna (medium) as its judge. The source report says the full benchmark contains 150 tasks, while Artificial Analysis ran the 100-task Main set.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Passing Checks Still Falls Short
The results show why passing tests alone may not establish that a patch is ready to merge. FrontierCode also penalizes excessive diffs, edits outside relevant files and tests that do not meaningfully establish correctness. Those criteria target the review burden maintainers face when assessing changes, not just whether code runs in a particular test suite.
For teams considering AI coding agents, the headline score is a measure of performance under this benchmark—not evidence that a model can independently produce acceptable changes in every repository. Even the highest-ranked model failed to pass all blocking checks in a sizable portion of attempts. Human review remains relevant, while results on this benchmark can help compare models under a shared set of tasks and grading rules.
Cost also changes the comparison. Artificial Analysis reported a per-task cost of $10.15 for Claude Opus 5.5 and $0.81 for GPT-6.1 Sol, with scores of 52.2% and 49.1%, respectively. The report describes the score gap as 3.1 percentage points and the cost gap as 12.5 times. Those figures reflect the tested setup and do not establish what a deployment would cost across other workloads.
As an affiliate, we earn on qualifying purchases.
A Benchmark Built Around Review
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026; version 1.1 followed in July. More than 20 maintainers created tasks drawn from 36 established repositories, including Celery, Budibase, uppy and Mattermost. The source report says contributors spent more than 40 hours on each task.
The tasks are private, according to the report, to reduce the chance that models encountered their solutions during training. FrontierCode also applies a trajectory scanner: attempts flagged for looking up solutions receive zero. Artificial Analysis added the benchmark to its evaluations on October 12, 2026, and published results for 23 models.
The source report compares these results with Artificial Analysis’s Intelligence Index and finds that some models shift position when evaluated on code quality and review criteria. For example, Claude Sonnet 5.5 ranks below GPT-6.1 Sol on FrontierCode despite its higher Intelligence Index score in the comparison. That contrast suggests a general model ranking does not by itself predict performance on this repository-focused evaluation.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
AI-powered code change approval software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Coverage and Real-World Limits
The scores describe performance on FrontierCode’s 100-task Main set, not every software project or coding task. The source report does not establish how closely benchmark outcomes predict acceptance rates across real-world repositories, where maintainers, workflows and risk tolerances differ. The grading also includes an LLM review step, and the reported results do not provide enough detail here to determine how sensitive rankings are to that judge.
Several models had no result as of October 13, according to the source: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the leaderboard is not a complete comparison of all models that might be relevant. It is also not clear from the material provided whether the reported per-task costs include every cost a user would incur in deployment.
software development code review tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
More Model Results May Follow
The next point to watch is whether results become available for models missing from the October 13 comparison, including Gemini 4 Argon and Muse Wonder Full. Further evaluations could change the ranking or clarify how performance compares across a broader set of systems.
For now, the published scores give developers a benchmark-specific comparison, not a recommendation to automate merges. Teams using the results would still need to test models against their own repositories and review processes. FrontierCode’s central finding remains limited but clear: the top tested model passed every blocking check in 57.7% of attempts under the reported evaluation setup.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated code changes using tests, command checks, limits on file and diff scope, and an LLM review against maintainer criteria. A failed blocking check gives the attempt a score of zero.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results, with a FrontierCode score of 52.2% and a 57.7% pass rate for all blocking checks.
Does the top score mean AI changes can be merged without review?
No. The result is a benchmark score, and the leading model did not pass every blocking check in 42.3% of attempts. The report does not establish that autonomous merging is reliable across real repositories.
How much did the leading models cost per task?
Artificial Analysis reported $10.15 per task for Claude Opus 5.5 and $0.81 for GPT-6.1 Sol in its tested setup. Costs may differ in other workloads or deployments.
Were all major models included?
No. The source report said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of October 13, 2026.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
