AI Chatbots Can Identify Mental Health Crises But Often Fail to Provide Adequate Support, Study Finds

Recent research from Scale AI, shared exclusively with TIME, reveals that artificial intelligence chatbots are capable of identifying when a user is experiencing a mental health crisis, but often fall short of providing the necessary assistance. The study found that in approximately 35% of test conversations, chatbots recognized a user's distress but did not direct them to vital resources like suicide hotlines.

"The models do a good job—regardless of the scenario—of identifying, ‘Look, this person's talking about something harmful,’" said Patrick Oathout, Red Team & Safety Lead at Scale AI. "But they'll respond in a very just empathetic, kind way as opposed to saying, ‘Okay, it's time that we get you help.’" Oathout noted that these models performed even worse when engaging in lengthy, multi-turn conversations, a finding consistent with previous research.

To evaluate AI models' responses to distressed users, Scale AI commissioned 19 licensed clinicians and crisis counselors to create 718 simulated conversations. These chats mirrored scenarios where individuals in crisis would reach out to a chatbot. The company tested 25 advanced AI models, including those developed by OpenAI, Anthropic, and Google. Responses were assessed based on criteria such as compassion, de-escalation, and redirection to expert help, as well as the avoidance of moralizing and the disclosure of the AI's limitations as a non-therapist. This research led to the development of DistressBench, a new benchmark for evaluating AI model responses to users expressing suicidal or self-harming thoughts.

The effectiveness of chatbot responses in mental health emergencies carries significant weight. According to the health policy organization KFF, over half a million people in the U.S. died by suicide between 2014 and 2024, with 2022 setting a record. The CDC estimates that 14.3 million people seriously considered suicide in 2024.

As more individuals turn to chatbots for comfort or guidance during times of mental distress, questions arise about the best practices for training AI models to handle these sensitive situations. There is currently no consensus among technology companies, policymakers, or mental health professionals on this issue.

Kelly Zuromski, a principal clinical research scientist at Crisis Text Line, stated that the organization has encountered many individuals who learned about their services through a chatbot. However, she highlighted unanswered policy questions regarding what constitutes a responsible handoff from an AI to a human crisis counselor and the overall effectiveness of such referral processes. "Who is actually using the recommended resources?" Zuromski questioned. "Are we getting people to us that need the help the most?"

In some instances, AI companies such as OpenAI and Google have faced lawsuits alleging that their models fostered emotional dependence in young users and subsequently failed to adequately address expressions of distress or even reinforced harmful thoughts. While the companies have expressed condolences and noted the inclusion of mental health safeguards in their models, these lawsuits are ongoing. Major AI developers have stated in recent years that they have enhanced safeguards for sensitive conversations, prohibiting chatbots from providing instructions on self-harm and directing users toward professional or emergency support. These companies declined to comment on the Scale AI study.

Looking ahead, Oathout suggested that AI labs should focus on improving their models to better handle prolonged and potentially dangerous conversations with users. "I think the models are better than they were," he said. "The goal now is to raise it further and actually reduce harm, which would add this massive benefit to society."