AI vs Doctor Can ChatGPT Diagnose Eye Disease

Could your next eye exam be read by an artificial intelligence? A recent groundbreaking study explored this very question, pitting the advanced AI model ChatGPT-4o against seasoned medical professionals in the specialized field of eye care.

The Study Pitting AI Against Experts

In a prospective diagnostic accuracy study, researchers set out to determine how well ChatGPT-4o could identify corneal diseases using only slit-lamp photographs. The AI was given a challenging task: to analyze 22 images sourced from the comprehensive ophthalmology collection at The University of Iowa's EyeRounds.org atlas and provide a diagnosis.

To create a true test of its visual analysis capabilities, the AI was given no additional clinical information, patient history, or context—only the raw image. For comparison, two consultant ophthalmologists were given the exact same task. The performance of both the AI and the human experts was then measured against the known, correct diagnosis for each image.

The Results How Did ChatGPT Perform

The findings revealed a fascinating picture of AI's current capabilities in medical diagnostics. ChatGPT-4o achieved a diagnostic accuracy of 50%. In comparison, the two human experts, referred to as Ophthalmologist A and Ophthalmologist B, scored 64% and 55%, respectively.

Statistical analysis showed that the performance gap between ChatGPT-4o and Ophthalmologist A was significant. However, there was no statistically significant difference between the AI's accuracy and that of Ophthalmologist B. This suggests that while ChatGPT-4o was not on par with the top-performing expert, its diagnostic ability was comparable to that of a consultant ophthalmologist.

What This Means for The Future of Ophthalmology

Researchers concluded that ChatGPT-4o demonstrates a moderate level of accuracy, highlighting the potential for large language models (LLMs) to serve as valuable adjunctive tools in ophthalmology. The study suggests that AI could act as a supportive aid, helping clinicians interpret images and make more informed decisions.

However, the study also pointed out a crucial limitation: the AI model sometimes produced confident-sounding responses that were incorrect. This tendency for 'confident inaccuracy' means that while the technology shows immense promise, it is not yet reliable enough for autonomous diagnostic use. The role envisioned for AI in the near future is one of collaboration, where it assists, rather than replaces, the expertise of a trained medical professional.

A Cautious Step Forward

This study marks an important step in integrating AI into clinical practice. While ChatGPT-4o is not ready to be a standalone diagnostician, its performance indicates a strong potential to support doctors and enhance patient care. For now, the most effective approach involves using these powerful AI models under the careful supervision of human experts, combining the best of artificial and human intelligence for better health outcomes.