What Researchers Tested Before Microsoft Said Its AI Beats Doctors

🔍 Read the full analysis: What Researchers Tested Before Microsoft Said Its AI Beats Doctors on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get smart everyday buys delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft reported that its MAI-DxO diagnostic system identified the correct diagnosis in 83.2% of 304 difficult medical case records, compared with 36.1% for generalist physicians using textbooks and the internet. The findings appeared in the peer-reviewed journal Science on October 22, but the test used historical, text-only cases rather than real patients, and compared the system with individual generalists—not specialists or clinical teams.

The paper, titled Sequential diagnosis with language models, reports results from Microsoft’s Sequential Diagnosis Benchmark. It included 304 complex cases drawn from the New England Journal of Medicine and a separate set of 325 emergency department cases. For the complex cases, the paper reports average diagnostic accuracy of 83.2% for MAI-DxO, 36.1% for physicians with reference materials and 22.3% without them. On the emergency department cases, the reported scores were 80.3% for the system and 46.8% for physicians.

The benchmark was designed to model diagnosis as a sequence of decisions, not a simple multiple-choice exam. Participants began with a brief case summary and had to request further information, such as a test result, from a simulated gatekeeper. The system’s architecture uses a panel of virtual physicians to propose diagnoses, choose questions or tests and challenge one another. The benchmark also tracked the estimated cost of tests along the way.

Microsoft’s figures are results from its own study, whose 15 listed authors were affiliated with the company, including Microsoft AI chief executive Mustafa Suleyman. Publication in Science means the paper underwent peer review; it does not show that an independent research group has reproduced the results. According to the supplied account of the study, GPT-5 on its own scored 78.9% on the difficult cases, close to MAI-DxO’s score. That suggests the base model accounted for much of the result, while Microsoft’s orchestration approach added a smaller gain and reduced estimated test costs.

Why the Doctor Comparison Matters

The study addresses a consequential question: whether AI can manage the step-by-step gathering of clinical information, rather than merely select an answer when all the facts are already provided. That capability could matter in settings where clinicians must decide which questions or tests to pursue. But a strong score on a selected case set does not establish that an AI system can safely diagnose patients in routine care.

The comparison group limits what readers can infer. The physicians were primary care doctors and hospital generalists working individually; the study did not test specialists or teams, despite the cases being unusually difficult. Real clinical decisions can also draw on physical examinations, imaging and other information that was not part of this text-based simulation. The findings therefore show performance under the benchmark’s conditions, not that MAI-DxO is better than doctors in hospitals or should replace them.

For patients, clinicians, hospitals and insurers, that distinction matters. Decisions about using diagnostic AI involve more than whether a system reaches a published diagnosis: they also depend on its performance across ordinary and complex cases, its ability to avoid harmful errors, and how it works alongside clinical staff. The results are a research finding, not evidence of an approved or routinely deployed diagnostic service.

How the Benchmark Worked

Conventional diagnostic benchmarks often provide a complete case and ask a participant to choose a diagnosis. Microsoft’s test instead started with an abbreviated presentation and required each participant to request additional information. A simulated gatekeeper held the full case record, and tests carried estimated prices. This was intended to capture both diagnostic accuracy and decisions about which information to seek and when to stop.

The supplied account draws some methodological details from a 2025 preprint, an earlier, non-peer-reviewed version of the research; those details may differ from the final paper. In that account, a model graded diagnoses against the published case answer, and physicians who re-graded a sample did not agree perfectly with the automated scoring. If a requested test result was absent from a case record, the gatekeeper generated a synthetic result. Test prices were estimates mapped to billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit charge.

The comparison also changed between versions of the work. The preprint described an earlier test in which doctors were evaluated on a smaller set of held-out cases, while the published paper reports physician results with and without reference materials. An earlier version, posted on arXiv in June 2025, reported a more dramatic gap using a different AI model. The published figures cited here are from the Science paper; earlier results should not be treated as identical measurements.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch

Limits of the Case-Record Test

The results do not establish how MAI-DxO performs with patients seeking care. The test relied on historical written cases, with no physical examination or real-world clinical workflow. The supplied source says the NEJM case set contained no healthy patients or harmless explanations, meaning the study could not measure how often the system would raise a false alarm in a broader population.

Other limits include the choice of physician comparison group and how performance was scored. The study did not include specialists or doctor teams, and the diagnosis grader’s judgments were not perfectly matched by physician reviewers in the earlier account of the methods. The test-cost figures were also estimates, not observed hospital spending. It remains unclear from the supplied material whether independent researchers will reproduce the scores or how the system would perform across a representative range of patients.

Planned Study Alongside Patient Care

Microsoft’s next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. The supplied source says consenting patients are expected to interact with the system alongside their usual care. That setting could provide evidence about how the tool operates during clinical encounters, but it would not by itself establish that it improves outcomes or is suitable for broad deployment.

The source does not give a start date, enrollment target, study design details or expected publication timeline. Until those details and results are available, the Science paper should be read as a benchmark comparison under defined conditions—not as proof that Microsoft’s system outperforms doctors in patient care.

Key Questions

Did Microsoft show that its AI is better than doctors in hospitals?

No. The paper reports higher scores on selected case-record benchmarks than for individual generalists under the study conditions. It did not test the system against specialists or clinical teams treating patients in hospitals.

What cases were included in the study?

The published results covered 304 complex cases from the New England Journal of Medicine and a separate set of 325 emergency department cases. Participants began with a brief summary and requested further case details.

What does the 83.2% score mean?

It is the reported diagnostic accuracy for MAI-DxO on the 304 complex case records. It is not a measure of the system’s accuracy across all patients or routine medical visits.

Was the result independently reproduced?

The supplied source does not report an independent reproduction. The paper was peer-reviewed and published in Science, but its listed authors were Microsoft-affiliated, and peer review is not the same as replication by an outside group.

What is expected next?

The source describes a planned prospective study at Beth Israel Deaconess Medical Center in Boston, where consenting patients would interact with the system alongside their usual care. Its timing and detailed design were not provided.

Source: ThorstenMeyerAI.com