Consumer AI models scored higher than specialized tools on medical knowledge test questions

A study published June 12 in Nature Medicine found that three widely available consumer AI models outperformed two specialized physician-facing tools on medical knowledge questions and clinical scenarios. Practicing clinicians graded the responses without knowing which system produced them.

On the MedQA portion, Gemini scored 97.4 percent, ChatGPT 94.2 percent, Claude 90.2 percent, OpenEvidence 89.6 percent and UpToDate 88.4 percent.

The consumer models cost $20 per month or less. UpToDate Expert AI requires a clinical subscription that can reach $600. The study tested OpenEvidence, UpToDate Expert AI, Claude Opus 4.6, GPT-5.2 and Gemini 3.1 Pro.

Forbes reported that the results challenge the assumption that expensive, narrowly built AI systems are required for high performance in medicine.S. adult users already turn to ChatGPT for health symptoms.

Chronic conditions such as diabetes, hypertension and heart disease affect 75 percent of Americans, while less than half of U.S. patients with hypertension have their blood pressure under control. Forbes reported that the FDA list of authorized AI-enabled medical devices now exceeds 1,500 entries.

Earlier narrow AI systems reduced false positives by 5.7 percent and false negatives by 9.4 percent in a U.S. mammography dataset evaluated in Nature. ChatGPT launched publicly in November 2022. Forbes reported that the study author, who teaches healthcare strategy at the Stanford Graduate School of Business, argues entrepreneurs should focus on tools that help patients use existing low-cost models rather than build high-priced systems sold only to hospitals.