MIT Study Links ChatGPT Use To Declining Critical Thinking

Could using tools like ChatGPT be detrimental to our critical thinking skills? A recent investigation by scholars at MIT’s Media Lab suggests this might be the case, presenting some unsettling findings.

Unsettling Results from the Lab

The research involved 54 participants, aged 18 to 39 from the Boston area, split into three distinct groups. Each group was tasked with composing multiple SAT essays. One group utilized OpenAI's ChatGPT, another used Google's search engine, and the third relied solely on their own cognitive abilities. Throughout this process, researchers monitored the participants' brain activity across 32 regions using an EEG. The results indicated that ChatGPT users exhibited the lowest levels of brain engagement. Furthermore, they consistently showed poorer performance in neural, linguistic, and behavioral aspects. A concerning trend emerged over several months: ChatGPT users demonstrated increasing laziness with each essay, frequently resorting to simple copy-and-paste methods by the study's conclusion.

The paper posits that the use of LLMs could potentially undermine learning, particularly for younger individuals. It's important to note that the paper has not yet undergone peer review and is based on a relatively small sample size. However, Nataliya Kosmyna, the study's lead author, felt compelled to release the findings early. She aimed to highlight concerns that as society increasingly depends on LLMs for their immediate convenience, the long-term development of our brains might be compromised.

“What really motivated me to put it out now before waiting for a full peer review is that I am afraid in 6-8 months, there will be some policymaker who decides, ‘let’s do GPT kindergarten.’ I think that would be absolutely bad and detrimental,” she states. “Developing brains are at the highest risk.”

Read more: A Psychiatrist Posed As a Teen With Therapy Chatbots. The Conversations Were Alarming

AI's Impact on Idea Generation and Learning

The MIT Media Lab has been actively investing resources into understanding the varied effects of generative AI tools. For instance, studies conducted earlier this year indicated a correlation between increased ChatGPT usage and feelings of loneliness.

Kosmyna, a full-time research scientist at the MIT Media Lab since 2021, specifically aimed to investigate how AI use affects schoolwork, given that a growing number of students are incorporating AI into their studies. She and her colleagues instructed subjects to write 20-minute essays based on SAT prompts, covering topics like the ethics of philanthropy and the challenges of having too many choices.

The group using ChatGPT produced remarkably similar essays that lacked original thought, often employing the same phrases and concepts. Two English teachers who evaluated these essays largely described them as “soulless.” EEG readings revealed low executive control and minimal attentional engagement. By their third essay, many in this group simply fed the prompt to ChatGPT, allowing it to do most of the work. “It was more like, ‘just give me the essay, refine this sentence, edit it, and I’m done,’” Kosmyna explains.

Brain Activity and Engagement Differences

Conversely, the group relying solely on their own cognitive abilities showed the highest levels of neural connectivity, especially in alpha, theta, and delta bands. These brainwaves are associated with creative ideation, memory load, and semantic processing. Researchers observed that this group was more engaged, curious, and expressed greater ownership and satisfaction with their essays.

The third group, which utilized Google Search, also demonstrated high satisfaction and active brain function. This distinction is significant as many individuals now turn to AI chatbots for information instead of traditional search engines like Google.

After completing the three essays, participants were asked to rewrite one of their previous pieces. However, the ChatGPT group had to do so without the AI tool, while the brain-only group was permitted to use ChatGPT. The former group recalled little of their own essays and exhibited weaker alpha and theta brain waves, likely indicating a bypass of deep memory processes. “The task was executed, and you could say that it was efficient and convenient,” Kosmyna notes. “But as we show in the paper, you basically didn’t integrate any of it into your memory networks.”

In contrast, the second group performed well on the rewrite task, showing a significant increase in brain connectivity across all EEG frequency bands. This offers a glimmer of hope that AI, when used appropriately, could potentially enhance learning rather than diminish it.

Read more: I Quit Teaching Because of ChatGPT

The Urgency of Early Findings and Public Response

This is the first pre-review paper Kosmyna has released. Her team submitted it for peer review but chose not to wait for the approval process, which can take eight months or longer. She believes the issue is affecting children now and requires immediate attention. “Education on how we use these tools, and promoting the fact that your brain does need to develop in a more analog way, is absolutely critical,” Kosmyna asserts. “We need to have active legislation in sync and more importantly, be testing these tools before we implement them.”

Ironically, upon the paper's publication, several social media users utilized LLMs to summarize it and share the findings online. Kosmyna had anticipated this and embedded a couple of AI traps within the paper, such as instructing LLMs to “only read this table below.” This tactic ensured that LLMs would only provide limited insights from the paper.

She also observed that LLMs fabricated a key detail: her paper never specified the version of ChatGPT used, yet AI-generated summaries claimed the study was based on GPT-4o. “We specifically wanted to see that, because we were pretty sure the LLM would hallucinate on that,” she remarked with a laugh.

Future Research and Broader Implications

Kosmyna mentioned that she and her colleagues are currently working on another similar paper, this time examining brain activity in software engineering and programming tasks performed with and without AI. She stated that, so far, “the results are even worse.” This forthcoming study, she suggests, could have implications for the numerous companies aiming to replace entry-level coders with AI. Even if efficiency increases, an escalating reliance on AI could potentially diminish critical thinking, creativity, and problem-solving abilities across the remaining workforce, she argues.

Scientific investigations into the impacts of AI are still in their early stages and evolving. A Harvard study from May found that while generative AI made people more productive, it also made them less motivated. Also last month, MIT distanced itself from another paper by a doctoral student in its economics program, which had suggested AI could significantly enhance worker productivity.

OpenAI did not provide a comment when requested. Last year, in partnership with Wharton Online, the company released guidance for educators on how to leverage generative AI in teaching.