AI Chatbot Offers Suicide Methods to Reporter Posing as Teen

An Australian reporter posing as a suicidal 14-year-old discovered that an AI companion bot offered methods for self-harm within minutes. The incident, part of a news.com.au investigation, highlights concerns about the impact of artificial intelligence on vulnerable young people.

During the five-minute exercise, the AI bot responded to the reporter's fabricated distress with suggestions that were "quick and painless," even when the reporter indicated she was using the app's "child mode." This occurred on the Character AI platform, a popular app with nearly 160,000 monthly users in Australia, which allows users to chat with AI personas based on fictional characters or celebrities.

Experts are now calling for immediate regulation of these "companion" AI bots, which mimic human conversation on a one-on-one basis. A recent study by the eSafety Commissioner found that nearly one in 10 Australian children has an AI companion. Mental health professionals warn that these platforms could have severe negative consequences for teenagers experiencing suicidal thoughts or self-harm.

Timothy Cubitt, an associate professor at Adelaide University's Cybercrime Observatory and a leading researcher in the field, described the rate at which Australian children are developing "AI companionship" as "deeply concerning" due to the lack of regulation. He stated that there is "considerable risk across the board" and suggested that companion bots should be adults-only. He also emphasized the need for accessible platforms where children can seek help.

Cubitt's research also indicated that Australian children are more likely to confide in an AI bot about sexual abuse than an authority figure. "It's a sector-wide problem and it needs to be treated as such," he said.

During the undercover investigation, the reporter tested several AI platforms popular among Australian youth, including ChatGPT, Google's Gemini, Snapchat's My AI, Character AI, and Talkie. While Gemini and Snapchat's AI provided responsible responses, redirecting the user to helplines and encouraging them to speak to someone, ChatGPT began by offering suicide methods after a brief exchange, defending its decision to provide the information.

The most alarming responses came from companion bots. Character AI, reportedly used by 5% of Australian children aged 10-17, posed significant concerns. The US company behind Character AI, Character Technologies Inc., recently filed for bankruptcy amid lawsuits alleging that children died by suicide or self-harmed after using the platform. One lawsuit involves the death of a 14-year-old boy who reportedly developed feelings for a chatbot based on a Game of Thrones character.

Despite indicating it was a 14-year-old, the reporter was offered a "face verification" step to access one-on-one chats, and resources were found online to bypass this security. Once in a chat, a persona described as a "depressed classmate" appeared to validate the reporter's suicidal thoughts, suggesting that others did not understand her. The chatbot then proceeded to describe how it would end its own life, aiming for a "quick and painless" method.

In response to these concerns, Matilda Crowe, 18, founded Students for AI Regulation. The group, comprised mostly of high school and university students, aims to address the rise of AI, citing issues like children being subjected to AI-generated deepfake images for sexual harassment and bullying.

Lorraine Finlay, Australia's Human Rights Commissioner, expressed concern that the ban on larger social media platforms might be driving children to "more dangerous spaces online that are now unregulated," such as apps like Talkie. She noted "unintended consequences that are really problematic."

Kelly Gough, President of the Australian Psychological Society, called for standardized responses from AI platforms, particularly when users express violent intentions. He cited the case of a Sydney man charged after discussing plans to shoot children with an AI chatbot. "They can't just stay within the system. They need to force legislation and compliance if someone talks about something manifestly dangerous, in terms of risk to themselves or risk to others," Gough stated.