Here’s how the FDA wants to regulate AI

The U.S. Food and Drug Administration is reviewing the future of medicine and wants your opinion on how it should handle machine learning. The agency, which has been ahead of the curve with its thinking about connected devices and AI in medicine, has issued a report and with it, a request for feedback on the topic of regulating machine learning.

It’s essentially asking whether or not devices that continuously learn and change their algorithms based on new data should be evaluated under a new set of rules to ensure that those algorithm changes don’t compromise patient safety. Put another way, if a medical device learns and adapts treatment based on that learning, how does the agency make sure it’s learning correctly, and in a way that won’t harm patients?

The full report is here and input is due June 3.

From a regulatory standpoint, machine learning is a tough one because in many cases it’s unclear how machine learning algorithms weigh information. Often it’s the outcomes themselves that we have to evaluate, but those outcomes will vary based on the individual and the amount of weight given to the individual’s data. And if the outcome is poor, it might be too late to fix the algorithm.

With that in mind, the FDA wants to look at several aspects of what it calls “Software as a Medical Device,” or SaMD. Among them are the risks and rewards of using a SaMD, which it aims to determine by asking questions such as: Is the device designed to help manage a condition? Is it designed to monitor overall wellness? Depending on the influence a particular device has on patient care, the FDA will hold the device to a higher standard. Diagnostic or management SaMDs will be held to the highest of all possible standards.

The FDA will also look at how the device evolves. Any device that uses machine learning and plans to adapt over time will be looked at to see how the machine learning algorithms it contains will change over time. The FDA recognizes three ways such a device may adapt.

The first is by changing its performance without changing either the data going into it or the way it will be used. For example, new data over time may help a software program get better at spotting breast cancer. So both the use case and the patient population stay the same, but the scanner improves.

The second is when a machine adds new inputs to address the same problems. So for example, a monitoring device might start incorporating heart rate data as part of an algorithm designed to determine wellness.

The third involves a change in the use case thanks to better ML algorithms. For example, perhaps a scanner has traditionally been awesome at detecting breast cancer in white women, but thanks to new data it can now accurately detect breast cancer in African-American women, too. The FDA considers a new patient population or a new diagnostic ability to be a change in the use case. It would also consider new weight given to different inputs as falling under the use case change category.

The agency wants to know if these classifications make the most sense and how it should ensure that changing input or the weight given to inputs remains safe for patients. The next area that concerns the FDA is how to make sure any algorithm adjustments are tested and then executed. The FDA doesn’t want companies to have a culture of “moving fast and breaking things” when it comes to building algorithms for diagnostics or patient care. And it plans to offer a good approach to machine learning, as laid out in the image above.

The goal is to ensure there are ways to measure the outcomes of algorithms over the lifetime of the product and that any company that makes a software-based medical device doesn’t just fling the product over the wall and forget about it. So instead of testing a drug and releasing it, a software-based medical device incorporating machine learning would have continued testing — the results of which would be transparent to regulators and consumers alike. The FDA wants to know if this is a good approach, as well as how medical device companies would have to restructure their testing operations to make it work.

After explaining in the report how it views the challenge of regulating machine learning in medical devices, the FDA tries to explain how it might handle various products, ranging from intensive care unit management software to a smartphone app that detects skin cancer. It then lays out what types of changes would trigger a new review and why.

As a consumer of medicine, I like that the FDA will ensure any changes in the incoming data and expansion of use cases will trigger a more comprehensive review. I also like that the agency wants device makers to show it their data as well as how they annotate and handle that data, and that device performance is monitored over time.

These rules will trigger changes in how devices are brought to market. It will also create new burdens for companies in the medical space. But I think it’s important for a regulatory agency to share how it’s thinking about the changes that connectivity brings to devices. As we add software and self-learning algorithms in more places (such as cars or even lighting systems), companies and other agencies should view the FDA’s initiative as a good example to follow.