AI And Rare Diseases How Accurate Is ChatGPTs Advice

The Challenge of Finding Information on Rare Diseases

Post-Orgasmic Illness Syndrome, or POIS, is a rare and often debilitating condition that causes systemic and cognitive symptoms after ejaculation. For patients dealing with such an underrecognized condition, finding specialist care and reliable, evidence-based educational resources can be extremely difficult. In this information vacuum, many are turning to artificial intelligence tools like ChatGPT for answers. This growing trend makes it crucial to evaluate whether the health information provided by generative AI is accurate, consistent, and easy to understand.

Putting AI to the Test A New Study on ChatGPT

To address this, a recent study assessed the performance of ChatGPT version 4o on questions related to POIS. Researchers selected sixteen real-world questions that patients might ask, dividing them into four key areas: epidemiology (how common the condition is), treatment options, risks associated with treatment, and counseling.

To test for consistency, each question was submitted to ChatGPT-4o on two different days from separate accounts. The responses were then independently graded for accuracy by three English-speaking urologists who are experts in men’s sexual health. The grading used a simple 4-point scale ranging from “correct and comprehensive” to “completely incorrect.” Researchers also analyzed the readability of the AI's answers to see if the language used was accessible to the average person.

How Did ChatGPT Perform The Results

ChatGPT-4o demonstrated a mixed performance. For questions about epidemiology and counseling, the AI was outstanding, achieving 100% accuracy and perfect reproducibility. This suggests it is quite capable of providing correct general information about the condition.

However, when it came to the critical areas of treatment and potential risks, the AI's performance dropped sharply. Accuracy fell to just 50%, and the answers were not consistently reproducible, meaning a patient could get different advice on different days. Perhaps most surprisingly, the readability of the responses worsened significantly from the first day to the second, with the language becoming more complex and less accessible.

The Verdict A Helpful Tool Not A Doctor

While ChatGPT-4o shows potential as a tool to support patient education for rare conditions like POIS, its unreliability in providing treatment-related content and its increasingly complex language limit its use as a stand-alone medical resource. These findings underscore a critical message: AI-generated medical advice requires expert oversight. Before large language models can be safely integrated into communication with patients, further refinement is needed to ensure they are consistently accurate, safe, and clear.