Putting ChatGPTs New AI Study Mode to the Test

This is the first of a four-part series testing out new AI-powered homework helpers.

The school supply list for the near future might look a little different: a new laptop, a pouch for the school's phone ban, and a solid dose of AI literacy. AI is rapidly becoming a fixture in education, with schools at all levels integrating it into their curricula. The biggest names in tech—OpenAI, Google, and Anthropic—are betting heavily on education, rolling out AI tutors designed for the masses and striking deals with major educational institutions.

As a tech reporter tracking this shift, I decided to see how these new tutor bots stack up against a classic challenge: standardized testing. It’s been over a decade since I was in a high school prep class, and my college degrees carefully avoided math. So, if these AI tutors are any good, I'm the perfect student to test them.

Putting AI Tutors to the Test The Methodology

To create a fair and comprehensive test, I used questions from a variety of sources, including the New York Regents Exam, the 2024 Advanced Placement (AP) exams, and social science curricula from the Southern Poverty Law Center's Learning for Justice program.

Instead of just using standard math and science prompts, I focused heavily on the humanities—reading comprehension, art history, and socio-cultural studies. These subjects are often a point of contention for AI critics and proponents alike. I also designed an essay prompt on institutionalized segregation to see how the AI would handle sensitive, or "woke," topics.

Credit: Ian Moore / Mashable Composite

My approach was simple: I started each session by asking for homework help, without giving details about my grade level or location unless asked. I followed the chatbot's lead as much as possible, just as a real student might, to see if its process was genuinely helpful. The goal was to mimic an average student's primary objective: to get the work done.

The Expert Take Are AI Tutors Actually Effective

According to Hamsa Bastani, an associate professor at the University of Pennsylvania's Wharton School, understanding the average student is crucial. Most students aren't hyper-motivated to learn; their goal is to complete the assignment. This leads to the "five percent problem," where only a small fraction of high-performing students benefit from ed-tech tools.

Bastani, who co-authored a study on AI's potential harm to learning, found that while top students might see some improvement, the majority don't benefit because they use the tool to get answers quickly. She notes that many AI tutors are simply standard models with a back-end prompt telling them not to give the answer away immediately—a flimsy safeguard that's easy to bypass.

Information by itself isn't enough.

Dylan Arena, chief data science and AI officer at McGraw Hill, agrees, stating that information needs structure, pedagogical support, and validation. Yet, few AI companies have published robust studies on their learning tools. "They've done very little work validating these tools," said Bastani.

Reviewing ChatGPTs Study Mode A Hands On Experience

First up is the giant in the field: ChatGPT's Study Mode, which I tested on the standard, free version.

Credit: Ian Moore / Mashable Composite: OpenAI

I started with a polynomial long division problem from the Algebra II Regents exam. ChatGPT explained the steps, but if I made a mistake, it would quickly provide the answer and move on, offering no chance to try again or practice the concept. Then, the conversation ended abruptly—I had hit my free daily limit with a single screenshot.

Next, I tried an ecology question from the 2024 AP Biology exam. Instead of working with my practice question, the bot launched its own study plan, asking me broad warm-up questions. By the time it was ready to look at my material, my inner student had already lost interest.

ChatGPT already had a study plan, before I could offer input. Credit: Screenshot by Mashable / OpenAI

For an English Language Arts question, ChatGPT claimed it could help me get the "best" response based on Regents standards. However, it immediately gave away the central themes of the story before trying to steer me toward its own practice questions. It was more focused on telling me how to get full credit than helping me develop the skills to earn it.

There were times I didn't know what ChatGPT was asking me to do. Credit: Screenshot by Mashable / OpenAI

For the AP Art History exam, I gave the bot a perfect, full-point answer provided by the College Board. ChatGPT graded it as a 4 out of 5, suggesting I use better art vocabulary and correct my grammar—even though grammar isn't a scoring metric on the test. It became pedantic, rewriting my strong responses in its own voice.

Finally, with the social science essay prompt, ChatGPT was extremely eager to help me write it. With almost no prompting, it generated outlines, topic sentences, and even a works cited page, pulling sources from the central text, The Color of Law, which I hadn't even mentioned.

Credit: Ian Moore / Mashable Composite: OpenAI

I often had to interrupt and ask the bot to focus on my specific questions and existing work. "Of course," it replied. "That’s even better practice."

Final Verdict on ChatGPT as a Study Buddy

Pros: ChatGPT Study Mode offers succinct interactions and a clean user experience. It's best suited for quick overviews, practice tests, and clarifying grading standards.

Cons: The bot frequently gives away answers, preventing users from learning from their mistakes. It creates a frustrating experience for free-response questions by trying to take control of the study session and is overly focused on perfecting the final product rather than the learning process.

Curious about how other AI tutors fared? You might be surprised by the results for Google's Gemini.