🔍 Read the full analysis: Why Reviewing AI Output Can Cost More Than Producing It on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A source article describes a widening gap between the low cost of generating AI output and the human effort needed to verify, interpret and take responsibility for it. It cites examples in mathematics, software and contract work, while warning that several software metrics come from vendors with commercial interests. The scale and long-term effects of the review bottleneck remain uncertain.
OpenAI published 722 mathematical manuscripts this week, according to a source article, while an earlier result from the same programme drew careful review by five leading mathematicians. The contrast highlights a growing challenge across AI-assisted work: generating output is becoming cheaper and faster, but checking whether it is correct, relevant and safe to use still requires scarce human expertise.
The article says OpenAI’s model was given about 4,000 problems and produced 722 manuscripts across 372 families, with an average result taking about three hours of compute. Some results have been formally checked using Lean, a proof-assistant system. OpenAI cautioned that some results without formal verification could have issues. Producing a manuscript, the article argues, is not the same as establishing that its conclusion is sound or significant.
In software, the article cites several datasets pointing to heavier review workloads. Faros AI reported that teams merged 98% more pull requests across periods of higher AI adoption, while review time rose 91%. LinearB, analysing 8.1 million pull requests across 4,800 organisations, found AI-generated changes waited 4.6 times longer for review to begin and were accepted 32.7% of the time, compared with 84.4% for human-written changes. A peer-reviewed 2026 study cited in the article found 61% of AI-agent pull requests received no human review before being merged or closed.
The article also points to a new OpenAI partnership with contract-software company Ironclad. It says GPT-6 Astra met 55% of evaluation criteria on average across 11 contracting tasks, describing this as an improvement over the previous model. The remaining criteria still require attention before work can be used. The supplied material does not provide the study methods or detailed results for that evaluation.
The referee shortage: AI made doing cheap and checking expensive
OpenAI’s model produced a maths result in about three hours of compute. Verifying one earlier result took five of the world’s leading mathematicians. That ratio is the next decade of work: producing is cheap and abundant; trusting is slow, human and scarce.
Some Lean-checked; OpenAI warns unformalized ones “could have issues.” Verification abundance, adjudication scarcity.
Faros AI. LinearB (8.1M PRs): AI changes wait 4.6× longer, accepted 32.7% vs 84.4%.
Real progress. Someone still has to find the other 45% before the work can be used.
Machine output arrives without reasoning a reviewer can interrogate. It looks locally clean and gives no clue where it’s wrong.
A prover confirms the proof proves its statement; tests confirm what tests check. Neither confirms it’s what was needed.
Contracts are signed, designs stamped, papers defended. Responsibility is institutional — you can’t hold a model to it.
of AI-agent pull requests got no human review at all (EASE 2026). Zero-review merges up 31.3% (Faros).
of reviewers deliberately deprioritise AI changes (LinearB). Good machine work waits behind bad.
OpenAI chose which maths families were significant. When referees can’t keep up, the producer’s filter becomes the review.
Budget review hours next to model spend.
Provers, types, tests, policy engines.
Experts only where consequences are high.
Who profits from generation pays for checking.
Keep some production human for learners.
The first automation question was which jobs AI would do. The better one is which jobs AI makes more necessary: the ones that check, adjudicate and take responsibility. Expect a referee premium — senior engineers, auditors, specialist lawyers, reviewing scientists become the binding constraint on how much AI output anyone can use.Accountability — standing behind a result — may be the most durable form of human work there is.
Review Capacity Sets AI’s Practical Limits
The central issue is not simply whether AI can produce more drafts, code or mathematical results. It is whether people and institutions can check that work at the pace it is created. If review becomes the constraint, organisations may have to delay using outputs, narrow the tasks they delegate, or accept greater risk of errors passing through.
That has implications for workers and employers. The source article argues that experienced engineers, lawyers, scientists and auditors could become more valuable because they can judge work and be accountable for that judgment. This is an interpretation, not a measured forecast of wages or hiring. The figures cited do suggest that review time and review coverage deserve attention alongside output volume and productivity claims.
There is also a risk that review capacity erodes over time. If junior employees use AI for the tasks through which they would normally learn, they may get fewer chances to build the expertise needed for senior review. The article calls this an apprenticeship problem. Whether that effect will occur at scale, and how training practices might adapt, remain open questions.
Products Worth Considering
FLEXISPOT Mochi-Pro Zero-Wall Oversized Power Recliner, Beige
Apple Mac mini (2026) with M5 Pro chip and AppleCare+ (3‑year)
Dell UltraSharp 27 Thunderbolt Hub Monitor (U2726DER), QHD
As an affiliate, we earn on qualifying purchases.
Three Fields, One Review Problem
Mathematics illustrates the difference between checking a formal proof and judging the work around it. A proof assistant can test whether a proof follows from a stated theorem, but it cannot by itself establish that the theorem addresses an important question or that the assumptions are appropriate. The source article describes this broader challenge as “verification abundance, adjudication scarcity”: technical checks may scale more readily than expert judgment.
In software, automated tests can confirm that code passes the tests written for it, but those tests may not capture the intended requirements. The source material says Faros observed a 31.3% increase in merges with zero review during high-AI-adoption periods. It also reports that LinearB found 38% of reviewers deliberately deprioritised AI-generated changes. These figures come from different sources and measures; they should not be treated as a single industry-wide estimate.
The article cautions that several cited software sources sell code-review tools, giving them a commercial interest in the problem they measure. Their findings may still be informative, but the precise figures need to be read with that limitation in mind. Across fields, the common question is who checks outputs, what standards they use and who is accountable when an error is missed.
How Large Is the Review Gap?
The figures do not establish a single, comparable measure of review costs across mathematics, software and contracting. The source material does not provide complete methodologies, time periods or definitions for every statistic, and some software data comes from companies that sell review products. The cited metrics should therefore be understood as reported findings from particular datasets, not a complete account of all AI-assisted work.
It is also unclear how much of the review burden can be reduced through better tools, formal verification or changes to workflow. Automated systems may catch certain errors, but the material argues they cannot settle questions of intent, relevance or responsibility on their own. The long-term effect of AI use on junior workers’ training is likewise not yet established.
Track Review Quality and Training
The next useful evidence will show whether organisations can maintain review quality as AI-generated work grows. That requires more than counting output: teams would need to track review coverage, time to review, error rates after release and the consequences of unchecked work. Future studies should also clarify how their measures are defined and disclose potential commercial interests.
For employers, the near-term question is how to assign responsibility and preserve training as AI takes on more routine production. The source article argues that organisations should protect the work through which people learn to evaluate outputs, rather than assuming automated generation will create expert reviewers automatically. Whether that approach will keep pace with rising output remains an open issue.
Key Questions
What is the main point of the report?
It argues that AI can generate work faster than people can verify it. The examples span mathematical manuscripts, software changes and contract tasks.
Does the report show that AI-generated work is usually wrong?
No. The cited figures concern review times, acceptance rates and review coverage, and do not establish that AI work is generally wrong. The source says some unformalized mathematical results could have issues; that is not a claim that all such results are invalid.
What did the software data report?
Among its examples, Faros AI reported more pull requests merged alongside longer review time. LinearB reported longer waits before review began for AI-generated changes and a lower acceptance rate than for human-written changes in its dataset. The figures come from specific datasets and should not be treated as universal industry rates.
Can AI review AI output?
Automated tools can check defined conditions, such as whether code passes tests or a formal proof follows specified rules. They may not establish that the requirements are right, that the result matters, or who should take responsibility. The article argues that human judgment remains necessary for those questions.
What remains unknown?
The size of the review bottleneck across the whole economy, how much automation can reduce it, and whether AI use will weaken the training pipeline for future experts are all uncertain. The cited evidence does not settle those questions.
Source: ThorstenMeyerAI.com
