Human Therapists Still Beat AI In Mental Healthcare

Key Insights from the Study

• ChatGPT demonstrated structure but often felt robotic and lacked nuanced empathy.
• Human therapists performed significantly better in six out of ten therapeutic domains.
• AI could still play a role in providing mental health care where access to traditional therapy is limited.

A human therapist outperformed ChatGPT-3.5 in delivering cognitive behavioral therapy (CBT) across all evaluated areas, as reported at the American Psychiatric Association Annual Meeting.

The Challenge of Meeting Mental Health Demand

"There is a growing demand for mental health care across the nation, but the number of providers isn’t necessarily there to meet that," Esha Aneja, BS, a fourth-year medical student at California Northstate College of Medicine, explained to attendees.

Although chatbots are now available, previous research has not examined their use in providing cognitive behavioral therapy. Image: Adobe Stock

Aneja noted that due to difficulties in accessing timely care, many patients are exploring AI platforms like ChatGPT.

Esha Aneja

"So, we were really curious about how AI can play a part in giving effective therapy," Aneja said. She mentioned that while previous studies showed benefits for mild depression using mobile app AI, they didn't specifically investigate CBT or modern large language models. "This is where we wanted to focus our attention," Aneja added.

How the Human vs AI Therapy Was Compared

The study involved a third-party patient who engaged with a human therapist via Zoom's chat function, with the therapist's camera turned off. Both the therapist and ChatGPT-3.5 (the latest version at the time) received the identical chat transcript. "We really wanted to have the same transcript given to the human therapist and the ChatGPT-3.5 and see really if there was a difference there,” Aneja stated.

Seventy-five mental health professionals and trainees, all knowledgeable about CBT principles, reviewed the transcripts from both human-patient and ChatGPT-patient interactions and completed a survey. The researchers then analyzed these survey results using the Cognitive Therapy Rating Scale (CTRS), which covers 10 domains, along with qualitative statistical analysis.

Clear Superiority of Human Therapists

"What was really surprising was that the human therapist outperformed the ChatGPT-3.5 across all of the domains,” Aneja revealed.

These differences were statistically significant (P = .001) in several areas:

• Feedback (Human: 4.43 vs. AI: 2.03)
• Collaboration (Human: 4.91 vs. AI: 3.84)
• Pacing (Human: 4.6 vs. AI: 3.67)
• Guided discovery (Human: 4.35 vs. AI: 3.45)
• Focus on key cognitions (Human: 4.61 vs. AI: 3.56)
• Application of CBT techniques (Human: 4 vs. AI: 3.69)

Aneja noted, "Ones that were more similar between the two groups were agenda setting, understanding, interpersonal effectiveness and strategies for change."

Furthermore, survey responses indicated a strong preference for the human therapist:

• 29% found the human therapist highly effective, versus fewer than 10% for ChatGPT.
• 52% gave the human therapist’s agenda-setting skills the highest rating, compared to 28% for ChatGPT.
• 24% rated the human therapist’s guided discovery highly, compared to 12% for ChatGPT.

The AI Experience Robotic and Impersonal

The qualitative part of the survey highlighted issues with empathy, rapport, structure, agenda, and personalization. "The most significant difference that people continuously pointed out with this analysis was that the AI sounded robotic and at times wasn’t able to give the nuanced empathy that the human interaction was able to give,” Aneja said.

Even though transcripts were blinded, some respondents identified ChatGPT's responses as AI-generated, describing them as generic and superficial. While many acknowledged room for improvement in human responses, the human therapist was generally seen as more natural and compassionate.

Respondents found ChatGPT’s approach structured and goal-oriented, but also robotic, with lengthy text blocks resembling a lecture. Interestingly, the human therapist was sometimes criticized for being too rigid and not allowing enough time for homework.

Adaptability was a challenge for both. ChatGPT’s advice often wasn't tailored to the patient’s specific situation. "Someone specifically wrote that they were seeking advice on a child custody situation, and the AI bot simply said, ‘use mindfulness,’ which did not seem appropriate,” Aneja shared. Similarly, one respondent felt the human therapist’s "strict adherence to the agenda sometimes led to missed opportunities to address the patient’s urgent needs, suggesting that more flexibility could enhance therapeutic relevance.”

Strengths of the Human Touch

The human therapist received praise for using metaphors, employing the teach-back method to check patient understanding, and building rapport. One respondent commented, "If I were this patient, I would think about the CBT triangle throughout the week, using it to reflect on how my thoughts and emotions interact.”

Aneja acknowledged that while ChatGPT showed understanding of the patient's internal reality and professionalism, its responses were often superficial and impersonal compared to the human therapist.

The Evolving Landscape of AI in Therapy

However, Aneja believes ChatGPT's empathy could improve. "This was ChatGPT-3.5, but now there are more updated models,” she said. "I’m sure that’s something that they’re going to work on. And I’m sure there’s also ways they can create it specifically tailored to psychiatry... I’m sure we’ll see it evolve in the future.”

She added that patients are already using AI chatbots for therapy. "I think, sometimes, patients consult it before maybe seeing a health care provider. Now that it’s so widely available, especially with the younger generations, more and more people are probably using it.”

Aneja anticipates increased AI use in therapy and across healthcare. ChatGPT could particularly benefit patients with limited access to therapy, such as those in rural areas, though with caveats. "Maybe they can’t get an appointment soon. It does do a good job understanding, but maybe keeping in mind some of the limitations of it, like the empathy component, could be important.”

She also suggested ChatGPT could serve as a screening tool when immediate provider access isn't possible. "If it’s done on a larger scale in psychiatry where it’s really tailored to looking for things like suicidality, certain principles that are really important to get urgent attention, that’s important. But I think it could definitely be used.”

Reference

• New research: Human therapists surpass ChatGPT in delivering cognitive behavioral therapy. View Source. Published: May 17, 2025. Published: May 17, 2025. Accessed: May 23, 2025.

For More Information

Source: Aneja E, et al. Evaluating the efficacy of AI-driven vs. human-delivered text-based cognitive behavioral therapy: A comparative pilot study. Presented at: American Psychiatric Association Annual Meeting; May 17-21, 2025; Los Angeles.

Disclosures: Esha Aneja, BS, can be reached at psychiatry@healio.com. Aneja reports no relevant financial disclosures.