How Nonsense Reveals ChatGPTs Unique Language Skills
Psycholinguist Michael Vitevitch, professor in the Speech-Language-Hearing Department at KU, sought to understand ChatGPT's internal workings by using experimental language prompts. (Image caption from original article)
A fascinating new study published in the peer-reviewed journal PLOS One, conducted by a psycholinguist at the University of Kansas, delves into how the popular artificial intelligence chatbot, ChatGPT, processes and responds to nonwords.
The Role of Nonsense in Language Research
Nonwords, which are essentially meaningless combinations of letters and sounds, are a common tool in cognitive psychology. Researchers use them to explore the intricate ways humans learn, remember, and process language. Michael Vitevitch, a professor in the Speech-Language-Hearing Department at KU and the author of the study, applied this established methodology to ChatGPT. His goal was to peek into the AI's "black box" and understand its language processing mechanisms.
"As a psycholinguist, one of the things I've done in the past is to give people nonsense to see how they respond to it — nonsense that's specially designed to get an understanding of what they know," Vitevitch explained. "I’ve tried to use methods we use with people to appreciate how they're doing what they're doing — and to do the same thing with AI to see how it's doing what it's doing.”
ChatGPTs Pattern Recognition A Different Approach
By feeding these nonsensical inputs to ChatGPT, Vitevitch discovered that the AI is quite adept at identifying relationships and patterns. However, a key finding emerged: "It finds patterns, but not necessarily the same patterns that a human would use to do the same task," he noted. This distinction is crucial.
"We do things very differently from how AI does things. That's an important point. It's okay that we do things differently. And the things that we need help with, that's where we should engineer AI to give us a safety net — to help us on those things,” Vitevitch elaborated, highlighting the potential for AI to augment human capabilities rather than merely replicate them.
Testing ChatGPT with Extinct Words
One part of the study involved testing ChatGPT's ability to define English words that have fallen out of use and are now effectively nonwords to modern speakers. An example is "upknocking," an 1800s occupation involving waking people up, a role now made obsolete by alarm clocks.
The results provided insight into the chatbot's internal workings. Out of 52 carefully selected extinct words, ChatGPT provided correct definitions for 36. For 11 words, it stated it couldn't recognize them. In three instances, the chatbot offered definitions derived from words in foreign languages. Interestingly, for two words, ChatGPT appeared to invent answers, a phenomenon often referred to as "hallucinations."
“It did hallucinate on a couple of things,” Vitevitch commented. “We asked it to define these extinct words. It got a good number of them right. On another bunch, it said, ‘Yeah, I don't know what this is. This is an odd word or very rare word that's not used anymore.’ But then, on a couple, it made stuff up. I guess it was trying to be helpful.”
Cross Linguistic Puzzles How AI Handles Sound Alikes
Another experiment presented ChatGPT with a set of Spanish words, asking it to provide similar-sounding words from English. This task mirrors the “phonological associate task” used with human English speakers to assess aspects of phonological similarity.
Vitevitch noted that this experiment particularly highlighted the difference in language processing between humans and AI. “If I give you a Spanish word and tell you to give me a word that sounds like it, you, as an English speaker, would give me an English word that sounds like that thing,” he explained. “You wouldn’t switch languages on me and just kind of give me something from a completely different language, which is what ChatGPT did.” He humorously compared this to multilingual parents code-switching to discuss topics without their English-only children understanding.
Rating Word Likeness AIs vs Human Intuition
In a third component of the study, nonwords designed to resemble real English words were presented to ChatGPT. The AI was then tasked with rating these words on a scale from one (“Bad English word”) to seven (“Good English word”), essentially assessing how much the nonword sounded like a plausible word in English. ChatGPT’s ratings were subsequently compared against those provided by human English speakers, offering another dimension to understanding its linguistic pattern recognition.
Can AI Create Sniglets ChatGPT as a Word Inventor
Taking inspiration from comedian Rich Hall's "sniglets" – humorous, invented words for concepts that lack a specific term – Vitevitch prompted ChatGPT to create new English words for novel concepts. Hall, on HBO’s “Not Necessarily the News” in the 1980s, famously coined terms like “carperpetuation” for that stubborn thread a vacuum cleaner repeatedly fails to pick up.
According to Vitevitch, the AI chatbot performed “kind of an interesting job there.” He observed that after being prompted for new words to match certain concepts, ChatGPT often relied on a predictable method of combining two existing words. One of Vitevitch’s favorites was “rousrage,” defined as anger expressed upon being woken.
Other word coinages generated by ChatGPT during the research include:
- “Prideify:” taking pride in someone else’s achievement
- “Lexinize:” the process by which a nonword begins to take on meaning
- “Stumblop:” to trip over one’s own feet
Understanding AIs Potential Through Its Quirks
By engaging ChatGPT with nonsense, Vitevitch aims to gain a deeper understanding of when and how AI can effectively assist humans in language-related tasks. The goal is not just to duplicate what humans can already do well, but to identify areas where AI’s unique processing style can offer genuine support and innovation.
