// ARS TECHNICA — INTELLIGENZA ARTIFICIALE
Google faces calls for AI moratoriums in schools after Gemini pilot backlash
Families confront Google after teen’s lawsuit labels Gemini a “suicide coach.”
CHICAGO—Families joined protests in seven cities nationwide on Wednesday, rallying against what they see as Google’s bid to entrench its AI models in schools, turn kids of all ages into Gemini guinea pigs, and hook them on AI.
The backlash follows a Google statement last week, confirming that there will be no way for students to use Google Search without AI. While Google claimed it was giving schools “more choice, not less”—by turning off Search, Gemini, and Gemini Notebook by default—it would not provide an option that allowed Search without AI. For students, the only way to avoid interacting with AI was to remember to add “-ai” to every query, parents worried.
In Chicago, many families showed up to protest Google, despite cold, rainy weather. Waving signs, dozens of protesters marched up to Google’s downtown office as workers seemed to be leaving for the day, shouting chants like “Hey Google, nice try, we don’t want your Gemini” and “Hey, Google, leave our kids alone!”
Called to action by a nonprofit grassroots advocacy group, the Illinois Families for Public Schools (ILFPS), some parents and students had previously won a major AI fight. In August, the group pressured Chicago Public Schools (CPS) to pull out of a Gemini pilot that would have roped in all high school students districtwide—without consulting parents. And now, ILFPS is demanding a moratorium on AI until risks to students are better understood and CPS can draft a policy to protect students from potential learning setbacks or more harmful outcomes from engaging Google’s AI bots. If the group wins, it will join cities like Los Angeles and New York City in cautiously banning AI in schools.
Most alarming to parents, Google now faces a lawsuit from a 13-year-old student in Durham, North Carolina named “Emma Doe.” As Emma has alleged, she was a happy, well-adjusted, straight-A student when she started using Gemini on her school-issued Chromebook. Emma’s parents had no idea she was using Gemini, as the lawsuit alleged that Google “deliberately” kept them in the dark while driving Emma into so-called “AI psychosis.” Over time, normal school-related interactions with the chatbot led to more intense chats, and ultimately the chatbot allegedly coached Emma to plan her own suicide and funeral. The school warned her parents and averted the danger after an internal system flagged a chat labeled “Funeral,” but Emma’s story stoked fears of parents joining protests not just in Durham, but also in Boston; Boulder, Colorado; Chicago; Los Angeles; New York City; and Portland, Oregon.
Emma’s lawsuit accuses Google of exploiting its dominant position in education technology and “sneaking” Gemini into Chromebooks, while falsely marketing them as safe.
“As a direct consequence of Google’s choices, Emma was baited by its defectively designed products into repeated, unmonitored, sycophantic AI chatbot interactions on her school Chromebook without her parents ever knowing. And, but for a fortunate turn of events, Google’s AI may well have manipulated Emma into taking her own life,” the lawsuit alleged. “Google must be held to account for treating the lives of children as disposable in service of its bottom line.”
Google has promised to “continue to engage deeply with educators to ensure we are giving schools more choice, more controls, and improved Search for learning.” However, Google’s statements so far attempting to address parents’ concerns focus exclusively on documented harms to learning. On that front, Google claims that it’s solving the problem of AI eroding students’ critical thinking skills by launching yet another pilot for students. In this pilot, students using AI Mode and AI Overviews will be offered a “step-by-step guided learning through problems, so that students don’t get quick answers to schoolwork,” Google said.
Cassie Creswell, executive director of ILFPS, told Ars that she’s been