Microsoft says its new health AI beat doctors in accurate diagnoses by a mile

Microsoft said its medical AI system diagnosed cases more accurately than human doctors by a wide margin.

In a blog post published on Monday, the tech giant said its AI system, the Microsoft AI Diagnostic Orchestrator, diagnosed cases four times as accurately as a group of experienced physicians in a test.

Microsoft’s study comes as AI tools rapidly make their way into hospitals and clinics, raising questions about how much of medicine can or should be automated and what role doctors will play as diagnostic AI systems get more capable.

The experiment involved 304 case studies sourced from the New England Journal of Medicine. Both the AI and physicians had to solve these cases step by step, just like they would in a real clinic: ordering tests, asking questions, and narrowing down possibilities.

The AI system was paired with large language models from tech companies like OpenAI, Meta, Anthropic, and Google. When coupled with OpenAI’s o3, the AI diagnostic system correctly solved 85.5% of the cases, Microsoft said.

By contrast, 21 practicing physicians from the US and UK — each with five to 20 years of experience — averaged 20% accuracy across the completed cases, the company added. In the study, the doctors did not have access to resources they might typically tap for diagnostics, including coworkers, books, and AI.

The AI system also solved cases “more cost-effectively” than its human counterparts, Microsoft said.

“Our findings also suggest that AI reduce unnecessary healthcare costs. US health spending is nearing 20% of US GDP, with up to 25% of that estimated to be wasted,” it added.

“We’re taking a big step towards medical superintelligence,” said Mustafa Suleymanthe CEO of Microsoft’s AI division, in a post on X.

He added that the cases used in the study are “some of the toughest and most diagnostically complex” a physician can face.

Suleyman previously led AI efforts at Google.

Microsoft did not respond to a request for comment from Business Insider.

Will AI replace doctors?

Microsoft said in the blog post that AI “represents a complement to doctors and other health professionals.”

“While this technology is advancing rapidly, their clinical roles are much broader than simply making a diagnosis. They need to navigate ambiguity and build trust with patients and their families in a way that AI isn’t set up to do,” Microsoft said.

“Clinical roles will, we believe, evolve with AI,” it added.

Tech leaders like Microsoft cofounder Bill Gates have said that AI could help solve the long-standing shortage of doctors.

“AI will come in and provide medical IQ, and there won’t be a shortage,” he said on an episode of the “People by WTF” podcast published in April.

But doctors have told BI that AI can’t and shouldn’t replace clinicians just yet.

AI can’t replicate physicians’ presence, empathy, and nuanced judgment in uncertain or complex conditionssaid Dr. Shravan Verma, the CEO of a Singapore-based health tech startup.

Chatbots and AI tools can handle the first mile of care, but they must escalate to qualified professionals when needed, he told BI last month.

Do you have a story to share about AI in healthcare? Contact this reporter at cmlee@insider.com.