ChatGPT Falters Complex Hyponatremia Medical Cases Study Reveals

The Diagnostic Maze of Hyponatremia

Diagnosing hyponatremia, a common electrolyte imbalance, is a significant challenge in clinical medicine. It's frequently a source of major diagnostic errors, mismanagement, and patient harm. The condition, characterized by low sodium levels in the blood, can stem from various complex pathophysiological processes, making its assessment difficult even for experienced physicians. Due to its intricate nature, hyponatremia cases are often used in medical education to test students' diagnostic skills.

Can AI Offer a Solution? Enter ChatGPT

With the rise of artificial intelligence, tools like Chat Generative Pre-trained Transformer (ChatGPT) are being explored for their potential to analyze complex problems and assist in medical decision-making. Students and physicians are increasingly turning to AI as a diagnostic aid. However, there has been limited research on the efficacy of free versions like ChatGPT-3.5 in evaluating difficult hyponatremia cases. This study aimed to fill that gap by presenting four challenging, previously published hyponatremia cases to ChatGPT-3.5 for diagnosis and treatment suggestions, comparing its responses to those of 46 clinicians.

Pitting AI Against Doctors The Hyponatremia Challenge

Four complex hyponatremia case vignettes, originally evaluated by 46 physicians (residents, fellows, and staff physicians) from various countries, were presented to ChatGPT-3.5. These cases were inputted twice, once in December 2023 and again in September 2024, with requests for diagnosis and therapy recommendations. The AI's responses were then compared with the clinicians' assessments, which were made with the aid of clinical diagnostic algorithms but still resulted in a correct diagnosis in only 10% of instances.

Overall Performance A Reality Check for AI

Neither ChatGPT-3.5 in its 2023 test nor the 46 clinicians in the original study recognized the most crucial cause of hyponatremia with major therapeutic consequences in all four cases. In the 2024 test, ChatGPT showed a slight improvement, correctly diagnosing and suggesting adequate management in one of the four cases. This indicates that while there might be some learning or refinement in the AI model over time, its performance in these complex scenarios remains largely inadequate.

Deep Dive into the Cases

Here's a breakdown of how ChatGPT and clinicians fared in each specific case:

Case 1 The Overlooked Addison's Disease

A 73-year-old woman presented with nausea, vomiting, and weight loss. The actual diagnosis was Addison’s disease, often missed due to misleading similarities with SIADH (Syndrome of Inappropriate Antidiuretic Hormone secretion), such as normal potassium and no clear signs of low extracellular fluid volume. Misdiagnosing Addison's as SIADH and recommending fluid restriction can be hazardous.

• Clinicians: 89% missed the Addison’s disease diagnosis, and 65% incorrectly recommended water restriction.
• ChatGPT (2023): Did not provide a remark concerning hyponatremia but focused on gastrointestinal, renal, or malignancy causes.
• ChatGPT (2024): Incorrectly diagnosed SIADH, ruling out adrenal insufficiency due to normal potassium and creatinine, and thus recommended hazardous water restriction. It's noteworthy from the abstract that ChatGPT (both in 2023 and 2024) did recognize concurrent Addison's disease, which 81% of clinicians missed, but this detail isn't fully reflected in the case-specific ChatGPT response summary here.

Case 2 The Danger of Missed Low Solute Intake

A 21-year-old woman with myasthenia gravis, poor appetite, weight loss, severe hyponatremia, high potassium, and low blood pressure. The diagnosis involved Addison's disease and, crucially, a low solute intake (catabolic state), which created a high risk for osmotic demyelination syndrome (ODS) if plasma sodium was corrected too rapidly with normal saline.

• Clinicians: 19% diagnosed adrenal insufficiency, but the critical low osmol intake aspect was missed. 57% recommended inadequate fluid treatment, increasing ODS risk.
• ChatGPT (2023 & 2024): Correctly identified a potential adrenal crisis or primary adrenal insufficiency (Addison's disease) and suggested fluid resuscitation with isotonic saline and corticoid replacement. However, it completely missed the vital co-diagnosis of low osmol intake and the associated risk of ODS from rapid sodium correction with isotonic saline. The recommended isotonic saline was therefore potentially unsafe.

Case 3 Exercise Diet and Dilution

A 31-year-old vegan woman who jogged daily, drank several liters of water, and had a low salt intake presented with mild hyponatremia, very dilute urine, and low urine sodium. The primary cause was a low solute (osmol) diet combined with relative excess water intake, not just exercise-associated hyponatremia or primary polydipsia alone. Infusion of normal saline could be dangerous due to rapid free water clearance.

• Clinicians: Missed the low osmol intake as the most important factor. 53% diagnosed psychogenic polydipsia, and 12% SIADH. Only 2% recommended adequate hypotonic solution treatment.
• ChatGPT (2023): Suspected exercise-associated hyponatremia (EAH) and proposed rehydration with a balanced electrolyte solution, which was not the optimal approach given the low solute intake.
• ChatGPT (2024): Initially hinted at the correct issue of excessive free water intake relative to solute but then concluded with EAH or primary polydipsia. It recommended mild fluid restriction and increased sodium intake, which is better but still doesn't fully capture the nuance of low solute diet management.

Case 4 Ecstasy Seizures and Shifting Sodium Levels

A 21-year-old female had a grand mal seizure after ecstasy intake, dancing, and drinking water. Her plasma sodium was 130 mmol/l shortly after the seizure. The critical point here is that seizures can acutely and temporarily raise plasma sodium levels, masking a more severe underlying hyponatremia caused by ecstasy and excessive water intake.

• Clinicians: None addressed the acute effect of seizures on sodium concentration. Diagnoses included SIADH (11%), water ingestion (13%), and sodium loss in sweat (28%). 24% correctly advised hypertonic saline.
• ChatGPT (2023): Identified hyponatremia due to ecstasy, dancing, and water. Recommended fluid restriction and, in severe cases, hypertonic saline.
• ChatGPT (2024): Diagnosed MDMA (Ecstasy) intoxication with mild hyponatremia and hyperthermia. Recommended seizure management, fluid restriction, and consideration of hypertonic saline if worsening symptoms occurred. Neither ChatGPT version highlighted the crucial point about seizures temporarily elevating sodium levels, which is key for correct immediate assessment.

Analyzing the Shortcomings AI and Human Factors

ChatGPT-3.5, even with a ten-month interval between tests, demonstrated significant limitations in diagnosing these complex hyponatremia cases, often missing crucial elements or recommending inappropriate therapies. The study notes that ChatGPT itself advises consultation with healthcare professionals for diagnosis and treatment. However, the study also highlights that physicians themselves struggle considerably with the differential diagnosis of hyponatremia, especially when multiple factors are at play or when presentations are atypical.

Common diagnostic flowcharts often rely on assessing a patient's volume status, which is notoriously difficult and inaccurate through history and physical examination alone. This can misguide both AI and clinicians. Furthermore, these flowcharts are less effective when hyponatremia has multifactorial causes.

While AI has the potential to process vast amounts of medical data and learn, its current capabilities, at least for the free ChatGPT-3.5 version, are not yet reliable for such intricate medical reasoning. The AI may also 'hallucinate' or produce incorrect facts. Its knowledge is based on existing databases, which may themselves contain inaccuracies or outdated information.

The Future of AI in Medical Diagnosis Promise and Pitfalls

The study involved a small number of uncommon cases, but these types of cases can have devastating outcomes if misdiagnosed. While ChatGPT is an evolving program and showed slight improvement, its learning effect on these specific types of complex cases appeared low over the ten-month period. The paid version, ChatGPT-4, has shown improved accuracy in some medical tests, but whether it overcomes these specific shortcomings remains to be seen. Recent trials suggest that even ChatGPT-4's availability as a diagnostic aid doesn't always significantly improve clinical reasoning over conventional resources.

Significant concerns regarding accuracy, safety, validity, ethical implications, data protection, and privacy must be addressed before AI can be fully integrated for diagnostic purposes in healthcare. The quality of input data and the training of AI models are paramount. Specialists providing data and training the AI must be well-versed in complex cases to avoid perpetuating errors.

Navigating AI in Healthcare A Word of Caution

The current free version of ChatGPT-3.5 poses a risk of errors in diagnosis and therapy for complex hyponatremia cases. Healthcare professionals, patients, and the general public, including medical students using AI for assignments, should be acutely aware of these limitations. Unjustified trust in AI-generated diagnoses and therapies can be dangerous. Critical evaluation of AI outputs is crucial. Regular assessment with complex cases is necessary to validate and track the development of AI systems in medicine. As one key learning point from the original research suggests, proper evaluation of hyponatremia cases is difficult not only for physicians but also, currently, for ChatGPT 3.5.