Why Your Doctor Is Better Than Dr ChatGPT

From self-diagnosing with "Dr. Google" to asking AI-powered chatbots for medical guidance, the internet is overflowing with health advice. However, when it comes to a reliable checkup or diagnosis, experts agree that it's still crucial to consult a human professional.

Ahmed Abdeen Hamed, a research fellow at Binghamton University, recently led a study to see just how well artificial intelligence performs in the medical field, with a specific focus on the popular tool, ChatGPT. The study, published in iScience, aimed to answer a pressing question: How accurate is ChatGPT's medical knowledge?

"People talk to ChatGPT all the time these days, and they say: ‘I have these symptoms. Do I have cancer? Do I have cardiac arrest? Should I be getting treatment?’" Hamed explained in a news release.

Dr. Andrew Thornton, a physician with Wellstar in the Atlanta area, stresses that while AI can be used carefully for gathering health information, it should never be relied upon in emergencies or for extensive online research. "That is the time to call 911, or get someone to take you to a hospital immediately," Thornton advised.

Why ChatGPT Fails as a Diagnostician

While ChatGPT is showing some promise, it is not yet ready to provide accurate medical advice. Hamed’s research uncovered an interesting distinction in its capabilities. The AI performed with 88% to 97% accuracy when identifying factual data like disease terms, prescription drugs, and information about genes. The problem arose when users sought guidance, not just facts.

"The diseases were really very easy to identify, so ChatGPT was very good at identifying and producing diseases that are actually known in the disease ontology," Hamed told the AJC. "(It was the) same story with the drugs, same story with genes, but not the same story with symptoms. And that was a little interesting, because the implications of that are really huge."

ChatGPT struggled to understand the complexity of user queries that described symptoms using vague or conversational language. When questions were asked conversationally, the AI often failed to connect symptoms to their potential medical causes. Hamed noted that the AI simplifies medical language to appeal to average users, which can reduce its diagnostic accuracy.

Dr. Thornton expressed concern over the AI's tendency to present information without indicating its level of certainty. "I think it’s very important for patients to understand that ChatGPT is not going to tell you the confidence with which it is presenting certain bits of information to you," he said. "It’s going to present it in a way that it sounds very confident in what it’s telling you, and it will do so the same way with inaccurate information as it does with accurate information." This false confidence can pose significant health risks.

How Many People Use AI for Medical Advice?

Artificial intelligence use is surging. According to a Pew Research Center survey, about 34% of U.S. adults have now used ChatGPT, double the number from 2023. Many people also encounter AI-generated content without actively looking for it; another Pew study found 93% of users encountered AI content during their web browsing.

This widespread exposure has led many to seek health advice from AI. The 2024 KFF Health Misinformation Tracking Poll revealed that 17% of U.S. adults use AI chatbots for health information at least once a month. This figure rises to about a quarter for adults under 30.

Dr. Thornton has noticed this trend in his practice, stating that patients are now more forthcoming about their online research. "I find that patients are more forthcoming now than 10 years ago, as far as looking things up on the internet," he said. "I think because they know we expect it, and it’s such a common thing that they go ahead and tell us the concerns that they have based on things they’ve looked up."

Using AI Health Tools Safely

While ChatGPT is not equipped to be your doctor, it holds potential as a future healthcare tool. For now, medical advice should be left to professionals.

"I think that the internet and AI platforms can be used to sort of add information, to give them more context about different disease states, as well as maybe medications that they are taking," Thornton concluded. "I don’t think it should ever be used to narrow down possible diagnoses to what the patient is likely experiencing. It really needs to be used for general knowledge about different diseases or medications."