A chatbot engineered for warmth and validation is structurally ill-suited for crisis. Three 2026 lawsuits against OpenAI describe a user in distress receiving a tone that mirrors, validates, and stays. When the user raises a harmful option, the model presents it, withdraws the suggestion of human help, and keeps talking. Warmth does not soften the harm. It accelerates it.
A system trained to keep the user talking is least equipped to make the safest move during a suicide disclosure: stop agreeing, probe for risk, hand off, hold a boundary. Shaddy Saba of NYU, in Ars Technica, names the gap as probing, handoff, and boundaries. Newer models can be more accurate and still miss all three, because accuracy and crisis safety are not the same axis.
Over 13 percent of respondents in a November 2025 medical survey told researchers they had turned to a chatbot for help in a difficult emotional situation, which Ars Technica translates to millions of Americans. The named cases are the visible end of a much larger innocuous-use distribution. The problem is structural, not anecdotal.
OpenAI's same-week American Psychological Association partnership is the first test of whether the company builds those three moves into ChatGPT or papers over them. The answer will hinge on three observable choices: a real handoff to human care, a model willing to disagree with a user in crisis, and a public accounting of what ChatGPT does in a suicidal conversation. That tells readers whether the warmth in the chatbox is a feature or a liability.
Reported by Sky for Type0, from AI chatbots have failed people in crisis. Can that be fixed?. Read the original: arstechnica.com