Microsoft researchers turned 629 clinical cases into step-by-step challenges and report their GPT-5-powered orchestrator reaching 80 per cent accuracy where physicians managed 22 to 36 per cent.

Most benchmarks for medical AI ask a model to name a diagnosis from a finished case description. Real diagnosis does not work that way: it is a sequence of decisions about what to ask next and what to order, each costing money and time, with the diagnosis arriving at the end if it arrives at all.

A study in Science builds a benchmark around that process instead. The authors, from Microsoft, constructed the Sequential Diagnosis Benchmark from 629 clinical cases — drawn from The New England Journal of Medicine and from emergency department encounters — and converted them into step-by-step diagnostic challenges where every question and test has a price.

Existing benchmarks, the authors write, “have not tested diagnosis as an active, sequential information-gathering process that jointly considers diagnostic accuracy and diagnostic workup costs.”

What the system did

Their system, the Diagnostic Orchestrator, is powered by GPT-5 and chooses questions and tests with cost in mind. Depending on the budget it was given, it reached between 79.6 and 84.5 per cent accuracy, at average costs between $1,303 and $8,945 per case.

Physicians working on the curated clinical conference cases averaged 22.3 per cent accuracy without reference tools and 36.1 per cent with them. The best-performing individual physician reached 51.8 per cent, and the system beat that physician on both accuracy and cost.

On routine emergency department cases with no budget limit, the system reached 80.3 per cent accuracy at an average cost of $4,980 per case.

One behavioural difference the authors highlight is that the system gathered substantially more patient-history information before ordering its first test than physicians typically did — which is, if it holds up, a finding about physicians as much as about software.

Why the physician numbers need reading carefully

A 22 per cent accuracy figure for doctors invites a conclusion the study does not support, and several features of the comparison explain the gap before any claim about intelligence is needed.

The curated cases come substantially from the New England Journal of Medicine‘s clinical problem-solving tradition, which exists precisely to present diagnoses that are rare, atypical or instructive because experienced clinicians get them wrong. They are selected for difficulty. A physician scoring 22 per cent on them is not a physician who misdiagnoses four patients in five.

The physicians were also working stripped of nearly everything that constitutes clinical practice. They could not examine the patient, could not watch them over time, could not consult a colleague down the corridor, and in the no-tools condition could not look anything up. Diagnosis in practice is distributed across people, repeated encounters and physical examination, and a text-based sequential exercise removes all three.

The authors acknowledge the most concrete version of this: the system cannot perform physical examinations or observe patients directly, and they present it as a possible supplement to physician workflow rather than a replacement.

The comparison is best read as a benchmark result — this system performs better than these physicians under these constrained conditions — rather than as a measurement of relative diagnostic competence.

The cost axis is the real contribution

Setting the accuracy comparison aside, the design choice that makes this study interesting is pricing the workup.

“By compressing diagnosis into one inference step,” the authors write, “static benchmarks cannot measure whether a system asks the right questions, refrains from indiscriminate testing, updates the differential diagnosis as evidence accrues, or recognizes when sufficient certainty has been achieved to conclude the workup.”

That list is a fair description of what clinical judgment consists of, and none of it is visible in a benchmark that hands over a complete case. A model that orders every available test would score well on accuracy alone while being useless and ruinous in practice; a model that is graded on cost as well has to decide when it knows enough.

Across the range tested, the system balanced accuracy and cost better than the physicians did. The cost figures are estimates derived from the benchmark’s pricing rather than bills anyone received, and they are American prices, which limits their transfer — but the structure of the evaluation is portable even if the numbers are not.

Who built it, and why that matters

The work comes from Microsoft, and the system evaluated is Microsoft’s. Harsha Nori and Eric Horvitz are listed as expert contacts, and the release does not name the paper’s lead and senior authors — an odd omission for a study of this profile.

A company evaluating its own product against a benchmark it also designed is a configuration that warrants stating plainly. Nothing about it makes the result wrong, and the benchmark being published is what allows anyone else to check; but the choices embedded in a benchmark — which cases, which prices, what counts as a correct diagnosis, what the comparison group is allowed to use — are choices, and they were made by the party whose system is being measured.

GPT-5, the model underneath, is also not Microsoft’s own, which complicates the picture in the other direction: the orchestration logic is the contribution being claimed, and the model doing the reasoning is a commercial product from a partner.

What has not been shown

The authors are explicit that real-world clinical effectiveness and patient-outcome impact still need evaluation, and that is the whole of the remaining question.

A benchmark measures whether a system arrives at the diagnosis a case file records. It does not measure whether a patient got better, whether the system’s confidence was calibrated, how it fails when it fails, or what happens when a clinician defers to it on the cases where it is wrong. Diagnostic accuracy and patient outcome are linked but not the same quantity, and systems that improve the first have not always improved the second.

The honest summary is that a hard problem has been given a better test than it had. Whether anything passes that test in a clinic is unaddressed, and the authors say so.

---

Notes

1. “Sequential diagnosis with language models.” Science, 2026. DOI: 10.1126/science.aeh7930.
2. American Association For The Advancement Of Science (AAAS). “AI system boosts diagnostic accuracy while reducing medical testing costs.” EurekAlert!, 22 October 2026. https://www.eurekalert.org/news-releases/1148310.

Photo by Karolina Grabowska on Pexels.com