Can ChatGPT Be Your Eye Doctor A New Study Investigates

The Rise of AI in Patient Health Inquiries

With at least 2.2 billion people affected by eye-related conditions globally, the internet has become a primary source for health information. Many patients research their symptoms online before ever stepping into a doctor's office. The emergence of powerful AI chatbots like ChatGPT introduces a new, popular resource for medical questions. However, the reliability of this AI-generated advice is a critical concern. A recent study aimed to evaluate how well ChatGPT performs when faced with common ophthalmology-related medical inquiries, focusing on its accuracy, comprehensiveness, and readability.

Putting ChatGPT to the Test A Rigorous Methodology

To assess ChatGPT's capabilities, researchers developed 22 questions based on the most commonly searched eye-related symptoms on Google Trends. These questions were then fed to the AI. The study employed a multi-faceted approach to evaluate the quality of the responses:

• Readability: The Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) formulas were used to determine how easy the text was to understand.
• Expert Review: Two board-certified ophthalmologists evaluated each response for accuracy, comprehensiveness, and clarity to gauge its overall appropriateness.
• Validated Tools: Other established quality metrics were used, including the QUEST and DISCERN tools, which assess the quality of consumer health information, and a custom urgency scale to see if the AI recommended appropriate action.

The Verdict A Mixed Bag of Results

The study's findings revealed a nuanced picture of ChatGPT's performance. On the positive side, all responses had an appropriate and supportive tone (QUEST Tone score of 2.0) and consistently recommended consulting a healthcare professional (Complementarity score of 1.0). The average scores for accuracy (4.41/5), comprehensiveness (4.89/5), and clarity (4.55/5) were high, with comprehensiveness being a particular strength.

However, there were significant weaknesses. Overall, the expert ophthalmologists rated only 68% of the responses as fully appropriate. The responses for glaucoma and cataracts—two serious, progressive conditions—received the lowest appropriateness ratings, highlighting a critical gap in the AI's reliability for complex diagnoses. Furthermore, the AI's ability to convey urgency was lacking. Only the response for narrow-angle glaucoma prompted the user to call for an ambulance, with the average urgency score being relatively low (2.45 out of a higher potential score). In terms of readability, the responses averaged a 10th-grade reading level, which may be too complex for many individuals seeking health information.

Key Takeaways Is ChatGPT Safe for Eye Advice

ChatGPT generally provides appropriate and well-supported responses to common ophthalmology questions. It excels in offering comprehensive information, maintaining a clear and helpful tone, and reinforcing the importance of seeking professional medical follow-up. However, its performance is inconsistent. The AI's weaker performance on critical conditions like glaucoma and cataracts is a major concern. While ChatGPT can be a useful starting point for gathering general information, this study underscores that it cannot and should not replace the expert diagnosis and personalized care of a qualified ophthalmologist.

About Channels

Unlock discounted publishing that highlights your organization and the peer-reviewed research and clinical experiences it produces.

• Learn more

Academic Channels Guide

Find out how channels are organized and operated, including details on the roles and responsibilities of channel editors.

• Learn more