Harvard Study Finds AI Models Excel in Some Emergency Care Reasoning Tasks Compared to Physicians

Artificial intelligence models outperformed physicians in emergency care medical decisions, according to a study conducted by researchers at Harvard Medical School and Beth Israel Deaconess Medical Center in the United States. The study compared AI and physicians across a wide range of clinical reasoning tasks.

Euronews reported these findings, highlighting the AI's superior performance in most experiments.

Researchers evaluated o1-preview, OpenAI’s reasoning model released in 2024. They provided the model with a range of clinical cases, including published case conferences and real-world emergency department records. AI outperformed human physicians across most experiments, especially in management reasoning, clinical reasoning, documentation, and real-world emergency settings with limited information.

In one evaluation, researchers asked the LLM o1 and GPT-4o to assess patients at various points in a standard emergency department setting, from early triage to later admission decisions. At each stage, the model received only the information available at that point and was tasked with generating likely diagnoses and recommending next steps.

The biggest gap between AI and human physicians occurred in the triage stage, where patient information is more limited.

Peter Brodeur, co-first author and HMS clinical fellow in medicine at Beth Israel Deaconess, said, 'Models are increasingly capable. The study reflects only model performance and primarily focuses on the preview version of the o1 model. The preview version of the o1 model has since been supplanted by newer models such as OpenAI’s o3 model.