AI Clinical Summarization Tools Sit In 'Regulatory Grey Area', Experts Warn

Clinical summarization large language models (LLMs) are AI tools that help clinicians generate and summarize medical records.

These tools sit in a “regulatory grey area,” Hugh Harvey, managing director at Hardian Health, said in a recent LinkedIn post.

Different jurisdictions have different rules governing the products, or sometimes none at all, said Harvey.

For example, while the US Food and Drug Administration (FDA) has established criteria to regulate clinical summarization tools under a non-device category, the EU has not issued any regulations defining the products.

However, Harvey noted in the LinkedIn post, the EU is notoriously cautious. “I would expect further clarity and guidance from the Medical Device Coordination Group,” he wrote.

Clinicians use clinical summarization tools as a time-saving alternative to notetaking. The tools record physician-patient interactions and collect patient data, then summarize the data to assist the clinician with diagnosis and treatment. The clinician can then review the notes to check for accuracy and add other details.

Last year, Google Research released MedLM, a group of generative AI models for “medical Q&A and summarizations.”  The product can be used in the creation of after-visit summaries or history and physical examinations. (Also see "News We’re Watching: GE And Google Announce AI Initiatives, Medtronic Announces Approvals, And More" - Medtech Insight, 8 Jan, 2024.)

Tech startup, TORTUS, has just secured $4.2m from Khosla Venture for generative AI model OSLER (Operating System Leverage in Electronic Records). OSLER processes physician’s notes in real time, providing a summary and the option for a patient letter.

EU MDR Regulation Of Clinical Summarization Tools

"It could be argued either way whether clinical summarization tools are medical devices or not when referring to the MDR medical device definition, Stephen Gilbert medical device regulatory professor, told Medtech Insight.

If a system is delivering diagnostic and/or therapeutic advice (a medical purpose) developers must clearly delineate the medical intended purpose and follow regulatory approval pathways for the system, said Gilbert. These systems are only medical devices if they go beyond simple database functions, said Gilbert. Systems which process input data through an algorithm to produce different output data, would be considered to be “beyond simple database functions”. (Also see "Chatbots Are Not Ready To Be Medical Devices, Experts Argue" - Medtech Insight, 7 Jul, 2023.)

Because clinical summarization tools are not actually doing the diagnosis, or deciding treatment, they are unlikely to be considered medical devices, however they are clearly part of the medical process, said Gilbert.

“In my view it is unlikely that an EU regulator would decide clinical summarization tools are part of the diagnosis or therapeutic process, and that they must be classed as a medical device,” Gilbert told Medtech Insight.

Lessons From The FDA

The FDA has adapted its clinical decision support software guidance to include a new “non-device” category.

To be considered a non-device, the software must meet four criteria:

• The software function does NOT acquire, process, or analyze medical images, signals, or patterns.

• The software function displays, analyzes, or prints medical information normally communicated between health care professionals (HCPs).

• The software function provides recommendations (information/options) to an HCP rather than providing a specific output or directive.

• The software function provides the basis of the recommendations so that the HCP does not rely primarily on any recommendations to make a decision.

Under this guidance, clinical summarization tools would fit the “non-device” category.

A recent Nature Portfolio Journal opinion piece from regulatory experts in Germany and France argued that the FDA’s non-device approach should be adopted by the EU, with careful post-market surveillance.

“If the FDA carries out careful market surveillance of the effects of this policy, then it may introduce enormous flexibility and innovation potential with minimal risk,” the paper said.

However, researchers from the University of Maryland School of Medicine are concerned that FDA guidance provides an “unintentional roadmap for how LLMs could avoid FDA regulation”.