Human Touch Wins in CBT AI vs Therapist Study

Study Pits AI Against Human Therapists in CBT

Fresh insights from the American Psychiatric Association’s Annual Meeting in Los Angeles reveal a comparative study on text-based cognitive behavioral therapy (CBT). The research pitted an AI therapist against a human counterpart, with human therapists demonstrating superior performance.

How the Research Was Conducted

The study involved 75 mental health professionals and trainees who participated in a cross-sectional survey. They assessed two text-based CBT transcripts: one generated by an AI and another by a human therapist. Evaluation was performed using the Cognitive Therapy Rating Scale, alongside qualitative feedback. Key CBT elements like agenda-setting, which involves outlining session tasks and ensuring their completion, and guided discovery, where patients explore personal data for self-learning, were scrutinized. To maintain consistency, both the human therapist and the AI evaluated identical clinical scenarios.

Key Findings Humans Outperform AI

The results clearly favored human therapists. Twenty-nine percent of participants rated human therapists as highly effective, a stark contrast to less than 10 percent for the AI therapist. In agenda-setting skills, over half (52 percent) of the participants gave the highest scores to human therapists, compared to 28 percent for the AI. For guided discovery, 24 percent of participants highly rated human therapists, while only 12 percent gave similar scores to the AI.

The Nuances AI Misses

While the AI therapist received comparable ratings in understanding a patient's internal reality, it was perceived as more rigid and impersonal. The researchers concluded that AI-based therapy is not suitable for standalone use but could potentially complement human-led therapy. Esha Aneja of the research team stated, "While our study found that AI-driven text-based cognitive behavioral therapy can apply basic therapy structures, human therapists outperformed ChatGPT-3.5 in key domains such as agenda-setting, eliciting feedback, and applying CBT techniques." Aneja added, "These findings underscore that although ChatGPT-3.5 shows potential, it currently lacks the nuanced empathy and therapeutic alliance that characterize effective human therapy."

About the Research Team

The research was conducted by Sebastian Acevedo, M.D., M.P.H.; Douglas Opler, M.D.; and Eric Jarmon, M.D., of Rutgers New Jersey Medical School, and Esha Aneja, MS4, of California Northstate University College of Medicine.

About the American Psychiatric Association

Founded in 1844, the American Psychiatric Association (APA) is the oldest medical association in the United States. It is also the world's largest psychiatric association, with over 39,200 physician members dedicated to diagnosing, treating, preventing, and researching mental illnesses. The APA's vision is to ensure access to quality psychiatric diagnosis and treatment. For more information, please visit www.psychiatry.org.