OpenAI Reasoning Model Outperforms ER Doctors in Clinical Diagnosis, Harvard Study Finds

A new study from Harvard Medical School and Beth Israel Deaconess Medical Center has shown that an OpenAI reasoning model outperformed experienced emergency room doctors in diagnosing and managing patient cases based on real-world emergency department records. Researchers tested the AI on messy, authentic clinical data, finding it superior in accuracy for complex tasks like generating diagnoses and treatment plans. As reported by Slashdot, the results highlight AI's potential to reshape clinical workflows, though they emphasize the need for careful prospective trials rather than outright replacement of physicians.

The experiment involved feeding the AI actual ER patient records, where it demonstrated stronger performance in critical thinking compared to doctors. According to details from the study, this builds on prior research, such as a 2024 analysis showing OpenAI models excelling at medical case studies, literature reviews, and patient scenarios. In one related trial summarized by cardiologist Eric Topol, AI alone achieved 92 percent diagnostic accuracy, surpassing physicians using AI assistance at 76 percent and those without it at 74 percent. Doctors often stuck to initial impressions, overriding correct AI suggestions and reducing overall accuracy.

LinkedIn cofounder Reid Hoffman, who now runs an AI drug discovery startup, has taken this a step further, arguing that failing to consult AI for medical advice is "bordering on committing malpractice." In a Wired interview, he advocated for doctors to routinely seek a "second opinion" from chatbots, positioning AI as an essential tool in modern healthcare. This view aligns with emerging evidence where AI independently handles tasks like interpreting chest X-rays or mammograms more effectively than human-AI teams in some cases.

The findings matter because they challenge traditional hierarchies in medicine, potentially speeding up diagnoses in high-stakes ER settings where time and accuracy save lives. Patients stand to benefit from fewer errors, while doctors could offload routine analysis to focus on nuanced judgment factors like a patient's insurance, physical limitations, or resource availability. Early applications, such as a Danish study where AI cleared half of normal chest X-rays or a Swedish trial triaging mammograms for 80,000 women, show real-world efficiency gains.

However, experts stress limitations: these are mostly retrospective studies with small samples, like one PMC analysis where GPT-4 beat ED residents against discharge diagnoses but called for larger validations. AI shines in generating broad differential diagnoses, including rare diseases, yet it lacks physical exams or real-time context. Researchers recommend prospective trials to test integration safely.

Looking ahead, the physician's role may evolve toward overseeing AI outputs, applying clinical expertise to refine suggestions into personalized plans. As Topol notes in his analysis, this "AI-first" approach—where models analyze data upfront—could transform diagnostics without sidelining human oversight. Broader adoption hinges on rigorous testing to build trust and ensure equitable access across healthcare systems.