Can AI Pass the Orthopaedic Surgery Exam

The Rise of AI in Medical Education

Artificial intelligence, particularly large language models (LLMs) like ChatGPT, are rapidly changing how we access and process information. While these tools have shown impressive results in passing complex legal and general medical licensing exams, their proficiency in highly specialized fields like orthopaedic surgery has remained an open question. For orthopaedic surgery residents, the Orthopaedic In-Training Examination (OITE) is a critical benchmark, measuring their knowledge and guiding their educational journey.

As trainees increasingly turn to AI for learning and information gathering, it becomes crucial to understand the accuracy and reliability of these tools. A recent study set out to evaluate ChatGPT's capabilities by pitting it against the OITE, providing a clear picture of its current utility as a didactic tool in orthopaedic training.

Putting ChatGPT to the Test

To gauge the AI's knowledge, researchers conducted a comprehensive test using 200 randomly selected questions from OITE exams administered between 2018 and 2022. The methodology was designed to be as fair as possible. Since ChatGPT cannot directly interpret embedded images, any questions containing visuals had the images uploaded to a hosting service, and a link was provided to the AI along with the question text.

The primary goal was to measure the percentage of correct responses. This performance was then rigorously compared against the scores of actual orthopaedic surgery residents, both from the researchers' local institution and from national averages across the United States. The study looked at performance across all post-graduate year (PGY) levels, from first-year residents (PGY1) to final-year residents (PGY5).

How Did the AI Perform

The results were clear: ChatGPT is not yet ready to outperform its human counterparts. The AI's scores were significantly lower than the national averages for allopathic orthopaedic surgery residents across all five PGY levels.

When compared to the local institution's residents, ChatGPT's performance was statistically similar to that of first and second-year residents (PGY1-2). However, more senior residents in their third (PGY3), fourth (PGY4), and fifth (PGY5) years of training all performed significantly better than the AI.

Interestingly, ChatGPT's performance varied slightly by exam year. It achieved its highest score on the 2021 exam, with 47.3% correct answers, and its lowest on the 2020 exam, with 35.3%. Despite this variance, the overall performance across the five years was not statistically different.

The Verdict on AI as a Study Partner

This study concludes that while ChatGPT can answer a portion of specialized orthopaedic questions correctly, its current knowledge base is roughly equivalent to that of a junior-level resident. It has not yet reached the proficiency of senior residents or achieved a passing score comparable to national averages.

The researchers suggest this performance gap is likely due to the model's limitations as a text-based tool that is not yet validated for nuanced medical image interpretation, a key skill in orthopaedics. However, they anticipate that future iterations of ChatGPT and other LLMs will undoubtedly improve.

For now, these findings provide crucial insight for medical educators and trainees. ChatGPT and similar AI models should be viewed as adjunctive educational tools rather than definitive sources of knowledge. As this technology continues to evolve, its role in resident education and assessment is expected to expand, but understanding its current limitations is essential for its responsible use in preparing for critical exams like the OITE.