ChatGPT Versus Google for Dental Pain Information

The Rise of AI Health Advisors

When a toothache strikes, turning to the internet for quick advice is a common first step. Increasingly, people are skipping traditional search engines and asking conversational AI tools like ChatGPT for health information. While this trend is growing, the reliability of AI-generated dental advice has remained largely unexamined, especially for a complaint as common as a toothache.

A new study sought to address this gap by evaluating the quality and readability of ChatGPT's responses to the most frequently searched questions about toothaches. The researchers aimed to see how the AI's answers stacked up against the information found on top-ranked health websites.

Putting ChatGPT to the Test

To conduct this cross-sectional study, researchers first identified the 20 most common toothache-related queries using a decade's worth of Google Trends data (January 2014-January 2024). They then presented each of these questions to the May 2024 version of ChatGPT in separate, independent sessions to ensure the AI's responses were not influenced by previous conversations.

Two expert endodontists were tasked with rating the quality of each response using the Ensuring Quality Information for Patients (EQIP) tool, a standardized measure for health information. The readability of the text was assessed using three different metrics: the Flesch Reading Ease score, the Flesch-Kincaid Grade Level, and the Simple Measure of Gobbledygook (SMOG) Index. To provide a benchmark, the researchers performed the same quality and readability analysis on the first 3-5 non-advertising websites that appeared in a Google search for each query.

The Verdict: ChatGPT vs. Google Search

The study's findings revealed that ChatGPT provides high-quality information regarding toothaches. The AI's responses achieved an impressive average EQIP score of 85.3 out of 100, with excellent agreement between the two specialist raters (Cohen’s kappa = 0.86; ICC = 0.91).

When compared directly to traditional websites, ChatGPT's content was found to be slightly superior in quality, scoring an average of 3.1 points higher on the EQIP scale. However, this higher quality came at the cost of readability. ChatGPT’s responses required a higher reading level, averaging an 8.4 grade level (Flesch-Kincaid), which was slightly more complex than the content from the websites.

What This Means for Patients and Providers

The study concludes that ChatGPT is a dependable source for toothache-related queries, offering information that is comparable, and sometimes even superior, to what can be found on top-ranked health websites. This positions it as a valuable tool for patients seeking initial guidance on dental issues.

However, the research also highlights a significant challenge in digital health communication. Both ChatGPT and conventional online sources present information at a moderate readability level, which could create a barrier for individuals with low health literacy. To ensure that digital health tools are equitable and effective for everyone, future efforts must focus on simplifying the language used in both AI-generated responses and website content to improve clarity and accessibility.