AI Chatbots vs. Physicians: What the Evidence Says About Diagnostic Accuracy

The Numbers: How Accurate Are AI Chatbots at Diagnosis?

The most rigorous analysis to date was published in Nature Digital Medicine in March 2025 — a meta-analysis pooling data from 83 studies on generative AI diagnostic performance.^[1] The headline finding: AI chatbots achieved an overall pooled diagnostic accuracy of 52.1%. To put that in perspective, that's roughly the accuracy of a coin flip.

The picture gets more nuanced when you break it down by comparison group. Against non-expert physicians (residents, general practitioners outside their specialty), AI performed comparably — the gap was a statistically insignificant 0.6%. But against expert physicians — specialists working within their field — AI models were significantly inferior, trailing by 15.8 percentage points (p = 0.007).

A separate study from the University of Virginia tested 50 physicians directly: half used ChatGPT Plus to diagnose complex cases, half relied on conventional resources like UpToDate and Google.^[6] The result? No significant difference in accuracy between the two groups (76.3% vs. 73.7%). The researchers noted that ChatGPT alone scored above 92% on the same vignettes — but when real physicians used it, the tool didn't actually improve their performance. The authors speculated that physicians may not know how to prompt AI effectively, or that the AI's confident-sounding outputs may create a false sense of certainty.

The Real-World Problem: AI Sounds Confident Even When It's Wrong

The Oxford study, published in Nature Medicine in February 2026, was the largest real-world evaluation of how the general public interacts with AI chatbots for medical advice.^[2] Nearly 1,300 participants were given medical scenarios and asked to identify conditions and recommend next steps — some using AI chatbots, others using traditional methods.

The findings were sobering. Patients using AI chatbots performed no better than those relying on a Google search or their own instincts. The core problem: chatbots produced a "blend of accurate and misleading information" that users couldn't reliably separate. The models that scored highest on standardized medical knowledge tests failed when interacting with actual humans in realistic scenarios.

Dr. Rebecca Payne, a co-author of the Oxford study and a practicing GP, was direct: "Despite all the hype, AI just isn't ready to take on the role of the physician. Patients need to be aware that asking a large language model about their symptoms can be dangerous, giving wrong diagnoses and failing to recognise when urgent help is needed."^[7]

Can You Trust an AI Symptom Checker?

Symptom checkers are a different product category from general chatbots. Apps like Ada, Buoy, WebMD Symptom Checker, and Symptomate ask structured questions about your symptoms and return a list of possible conditions plus a triage recommendation: stay home, see a doctor, or go to the ER. About 22% of US adults now get health information from AI chatbots at least sometimes, yet only 18% of adults rate that information as highly accurate.^[14] So how do these tools actually perform when researchers audit them?

The Accuracy Record: A Decade of Audits

The landmark audit tested 23 symptom checkers against 45 standardized patient cases. The correct diagnosis appeared first only 34% of the time, and triage advice was appropriate in 57% of cases.^[11] Follow-up audits have not moved the needle much: a 2022 systematic review across 10 studies found primary-diagnosis accuracy ranging from 19% to 37.9%, with triage accuracy between 48.8% and 90.1% and marked variability between tools given identical cases.^[9]

The Safety Myth That Quietly Reversed

For years the standard reassurance was that symptom checkers err on the side of caution. That was true in 2015, when they over-triaged at an odds ratio of 2.82 to 1. When researchers re-tested 22 of the same apps five years later, the ratio had collapsed to 1.11 to 1, and the apps' median sensitivity for true emergencies was just 51.9%. In plain terms, the 2020 generation of symptom checkers sent fewer people to care they did not need, but it also missed more than 40% of genuine emergencies.^[10] The "safely over-cautious" reputation is out of date.

Are ChatGPT-Style Tools Better Than the Apps?

Somewhat, on consistency. The most current evidence synthesis, published in npj Digital Medicine in 2025, compared symptom-assessment apps, large language models, and laypeople on self-triage. Apps ranged wildly from 11.5% to 90% accuracy depending on the tool; LLMs clustered between 57.8% and 76%; laypeople with no tools scored 47.3% to 62.4%.^[9] The authors concluded that neither apps nor LLMs can be universally recommended or discouraged.

Consistency is not safety. In an emergency-department study using 40 real de-identified patient cases, ChatGPT-3.5's triage advice was rated unsafe in 41% of cases, compared with 14% for the Ada app. The authors' conclusion was blunt: unsupervised patient use of ChatGPT for diagnosis and triage is not recommended.^[12]

Same Symptoms, Different Answers

Even the same tool can give different answers depending on who types. When three testers entered identical patient cases into popular symptom checkers, agreement between testers ranged from a kappa of 0.49 with free-text input to 0.72 with structured input, and diagnostic accuracy varied by 13 percentage points between the highest and lowest performing tester. Only 30.7% of cases were entered identically to the source case by all three testers.^[13] How you describe your symptoms changes the answer you get, which is precisely the variable a physician's structured history-taking is designed to control.

One more thing worth knowing: most consumer symptom checkers are not FDA-reviewed products. FDA's clinical decision support framework, finalized in 2022, technically treats software that gives recommendations directly to patients as a medical device, yet most consumer tools operate under separate low-risk general wellness policies instead. The regulatory status of patient-facing AI triage remains genuinely unsettled.^[17]

A reasonable way to use these tools: as preparation, not as a verdict. Write down what the checker suggests, then bring that list to a licensed clinician, whether that is a telehealth visit or an in-person appointment, and let a physician take the history that the app could not.

The Dependency Trap: When Physicians Stop Thinking Independently

There's a risk that cuts in the other direction, too. A study published in The Lancet Gastroenterology and Hepatology tracked gastroenterologists in Poland who used an AI system to detect polyps during colonoscopies.^[8] After just three months with the AI turned on, the physicians' detection rates dropped roughly 20% when the system was turned off. They had already begun relying on the AI as a safety net — and their independent diagnostic skills had started to erode.

A JAMA study demonstrated the flip side of the same coin: when physicians were shown intentionally biased AI predictions, their diagnostic accuracy dropped by 11.3% — even when they were given explanations for the AI's reasoning.^[5] The explanation didn't protect them from following the AI's lead. Physicians anchored to the AI output regardless.

This is what makes the question "Is AI better than doctors?" too simplistic. The real concern isn't whether AI can pass a medical exam. It's what happens to physician judgment when AI is embedded in clinical workflows without proper safeguards — and what happens to patients when they replace a trained clinician with a chatbot conversation.

What AI Cannot Do (Yet)

AI chatbots process text patterns. They generate statistically likely word sequences based on training data. What they do not do — in any current form — is clinical reasoning. Here's what that means in practice:

• They can't read your body language. A patient who says "I feel fine" while appearing acutely uncomfortable tells a clinician more than the words alone convey.
• They can't ask the right follow-up questions. In clinical experience, the questions a patient doesn't think to answer are often the most diagnostically important ones. A skilled physician recognizes what's missing from a history.
• They can't integrate context the way a clinician does. A 22-year-old woman with burning urination is a different clinical picture than a 72-year-old man with the same symptom — even though a chatbot might return the same first-line diagnosis for both.
• They have no accountability. If an AI chatbot misdiagnoses a pulmonary embolism as anxiety, there is no malpractice framework, no professional license at stake, and no one to call the patient back when the diagnosis doesn't sit right.
• They can't say "I don't know." Large language models are designed to produce an answer every time. They do not flag uncertainty the way a careful clinician does when a presentation doesn't fit a clean pattern.

Where AI Belongs in Healthcare

None of this means AI has no place in medicine. It does — but as a tool, not a replacement. The evidence consistently shows that the strongest outcomes emerge when trained physicians use AI to augment their judgment, not substitute for it.

A Stanford study found that physicians paired with AI chatbots performed as well as the chatbot alone on clinical management reasoning tasks — and both outperformed physicians without AI access. The key word is "paired." The physician was still in the loop, applying judgment, filtering the AI's output through clinical experience, and making the final decision.

That's the model that works. An AI system that surfaces relevant literature, flags potential drug interactions, or suggests differential diagnoses for a physician to evaluate is a genuine clinical asset. An AI chatbot that tells a patient with chest tightness that they probably have acid reflux — when a physician would order an EKG — is a liability.