Hi friend,

Generative AI is hard to regulate.

The systems are open-ended, probabilistic, and it’s hard to test them. To address these challenges in mental health, regulators are adapting their current medical device regulations to be suitable for AI systems (see our briefings on AI regulation in the US, Europe and the UK)

But what happens after a device is authorised? 

This is perhaps the hardest part of the regulatory process. AI models change. A company might update its system prompt, retrieval corpus, guardrails or routing logic. The foundation-model provider they use might ship a new version of their model. All of these could change how the device behaves. 

Even if the device stays the same, the world around it changes: language evolves and users behave differently. Because of the open-ended and generative nature of LLMs, new inputs can generate new outputs, and new failure modes can emerge.

In short, the product a regulator authorised may not be the product a patient encounters after authorisation.

There are two main approaches regulators are using to combat this challenge: Predetermined Change Control Plans (PCCPs) and the direct probing of deployed models. 

Neither is perfect.

PCCPs have received the most attention but face many challenges. Direct testing holds potential in ensuring AI devices work well in the real world but has problems of its own. In this report, our third instalment on AI regulation with May Lee, we explain both approaches, their benefits, drawbacks and some ideas about what needs to be done to ensure AI is safe and effective in mental health.