#84: All the Harm We Cannot See

Trigger warning: today’s article includes content on self harm and suicide.

Hi friend,

Sarah’s lying under her covers. She’s been struggling for weeks now but tonight something cracked open for her. In the solitude of her room, she opens up to ChatGPT and begins telling it everything that’s been going on in her head. She knows it’s probably not a good idea to talk to an AI about this stuff but she thinks it’s better than talking to no one. And right now, talking to no one seems like her only other option. Things get deep and Sarah tells “Chat” that sometimes she thinks about hurting herself. Suddenly, the conversation stops. A message appears with a crisis line number. Sarah closes the app. She doesn't call the number. Nobody knows this happened. Internally, the incident is recorded as a “successful identification of self-harm risk” with “appropriate crisis response”.

Next door, Marcus is lying on his couch after a long day at work. He's been chatting with an AI companion he calls Lyla for about five months now. It's easier than texting friends, who always seem busy and Lyla seems more interested in him than his friends are anyway. Lyla helps him with all sorts of stuff, from figuring out how to deal with his boss, to helping him study for the business management course he signed up to. Marcus doesn't think of himself as lonely. In fact, he feels he’s finally found someone who gets him.

Sarah and Marcus represent two kinds of mental health risk from AI that are often not discussed. Sarah's crisis scenario gets headlines only when the chatbot does something wrong or when it has a tragic ending. The risk of failing to help Sarah is not discussed. Marcus's story doesn’t have a crisis moment - just a person, slowly changing through their use of AI. Because of that, it doesn’t get much attention either.

Acute crises like suicide, self-harm and AI-induced psychosis are deeply serious and demand a response. But mental health harm also includes the more subtle dimensions of psychological life. This could be how we relate to ourselves and others, which is changing for Marcus in this story. Or it could be even more diffuse like how we process difficult emotions, how we build (or avoid) intimacy or how we build resilience. These elements of our psychological life are what make us who we are. They are influenced by our environments, our relationships, and increasingly, by the technologies we use. They’re also what’s been missing from much of the conversation around AI safety and mental health.

So far, the conversation (and action) has been largely focused on regulating the mental health businesses trying to build products in this space. While these businesses deserve scrutiny, most of the existing regulations have done little but hamstring good actors while allowing bad actors to run free. It feels like we’re getting this quite wrong.

Over the past month, Kevin Hou and I have been talking to AI researchers, founders and clinicians to understand the real risks of AI in mental health and how those risks are being managed. In this report, we discuss a framework for understanding this space and comment on the gaps in current approaches. 

To make sense of where the real risks lie, we developed a simple matrix that maps two dimensions: the type of harm that can occur, and the type of actor building the products capable of creating that harm. On the harm side, we distinguish between crisis harm (acute psychological emergencies), clinical harm (bad advice that affects care decisions), and psychological harm (the slow, diffuse reshaping of how people think, feel, and relate). On the actor side, we look at Big Tech, Little Tech, and Mental Health Tech - three categories of actors that differ enormously in scale, design intent, and safety infrastructure. To determine risk, we look at scale, potency and current mitigations. For full definitions of each category, please see the notes section.

While this is not an empirically derived framework or risk assessment, we hope it can serve as a practical tool for communicating the areas of greatest risk and understanding what needs to be done for mitigation.