The imperative for regulatory oversight of large language models (or generative AI) in healthcare.

Meskó B, Topol EJ. · 2023

Read The imperative for regulatory oversight of large language models (or generative AI) in healthcare. by Meskó B, Topol EJ. (2023) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc10326069.txt.

Introduction

Meskó and Topol argued in 2023 that large language models in health care need firm rules. Such models, like GPT-4 and Bard, can write text that sounds human. Clinics may use them to draft notes, sum up papers, or chat with patients. GPT-4 can also read text inside images.

Meskó warns that these tools learn in a new way, unlike older medical software. Their answers can vary in quality, which adds risk for patients. Poor use could harm people or leak private data. The authors set out steps for rule makers to take. Their aim is safe use that still lets the tools help care.

Catalog record

CREATORMeskó B, Topol EJ.
YEAR2023
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC10326069

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The imperative for regulatory oversight of large language models (or generative AI) in healthcare By Bertalan Meskó, Eric J. Topol Published in: NPJ Digital Medicine DOI: 10.1038/s41746-023-00873-0 ABSTRACT The rapid advancements in artificial intelligence (AI) have led to the development of sophisticated large language models (LLMs) such as GPT-4 and Bard. The potential implementation of LLMs in healthcare settings has already garnered considerable attention because of their diverse applications that include facilitating clinical documentation, obtaining insurance pre-authorization, summarizing research papers, or working as a chatbot to answer questions for patients about their specific data and concerns. While offering transformative potential, LLMs warrant a very cautious approach since these models are trained differently from AI-based medical technologies that are regulated already, especially within the critical context of caring for patients. The newest version, GPT-4, that was released in March, 2023, brings the potentials of this technology to support multiple medical tasks; and risks from mishandling results it provides to varying reliability to a new level. Besides being an advanced LLM, it will be able to read texts on images and analyze the context of those images. The regulation of GPT-4 and generative AI in medicine and healthcare without damaging their exciting and transformative potential is a timely and critical challenge to ensure safety, maintain ethical standards, and protect patient privacy. We argue that regulatory oversight should assure medical professionals and patients can use LLMs without causing harm or compromising their data or privacy. This paper summarizes our practical recommendations for what we can expect from regulators to bring this vision to reality. INTRODUCTION The rapid advancements in artificial intelligence (AI) have led to the development of sophisticated large language models (LLM) such as OpenAI’s GPT-4 and Google’s Bard1,2. The unprecedented popularity of ChatGPT, GPT-4’s predecessor released in November 2022, is reflected by the most rapid uptake of users - 100 million in 2 months - for any new technology. This rapid growth sparked global debates about the role such conversational chatbots could play in healthcare and the practice of medicine. Diverse applications of LLMs have appeared including facilitating clinical documentation; creating discharge summaries; generating clinic, operation, and procedure notes; obtaining insurance pre-authorization; summarizing research papers; or working as a chatbot to answer questions for the patients with their specific data and concerns. LLMs can also assist physicians in diagnosing conditions based on medical records, images, laboratory results, and suggest treatment options or plans. At the same time, patients can potentially become more autonomous than with prior search methods by obtaining individualized assessment of their data, symptoms, and concerns. Systematic reviews highlighted other potential benefits too such as improved scientific writing, enhancing research equity, streamlining the healthcare workflow, cost saving, and improved personalized learning in medical education3,4. Given the potential implications on patient outcomes and public health, it is imperative to consider how these new AI-based tools should be regulated. The regulation of these LLMs in medicine and healthcare without damaging their promising progress is a timely and critical challenge to ensure safety, maintain ethical standards, pre-empt unfairness and bias, and protect patient privacy. Whatever concerns have been previously recognized with AI are now markedly amplified with the multipotency of LLMs. This paper explores the potential risks and benefits of applying LLMs in healthcare settings and argues for the necessity of regulating LLMs differently than AI-based medical technologies that are already on the market to mitigate potential harm and maintain public trust in these breakthrough technologies. LLMS DIFFER FROM ALREADY REGULATED AI-BASED TECHNOLOGIES LLMs differ significantly from prior deep learning methods in terms of their scale, capabilities, and potential impact. Here we outline the key characteristics of LLMs that set them apart from traditional deep learning techniques. Scale and complexity LLMs are trained on massive datasets and utilize billions of parameters, resulting in unprecedented complexity. This level of sophistication requires regulatory oversight that takes into account the challenges associated with interpretability, fairness, and unintended consequences. Moreover, LLMs use tokens that can be words, subwords, or even characters as the smallest units of text used to represent and process language during the training and generation processes. Tokenization is a crucial step in natural language processing (NLP) and allows LLMs to efficiently analyze and generate text, as these models are designed to process sequences of tokens rather than entire sentences or paragraphs. Currently, tokenization is not covered by healthcare regulators. Hardware requirements LLMs require massive computational resources in terms of floating-point operations per second (FLOPs) and graphics processing unit (GPU) usage compared to previous deep learning models due to their large scale, extensive training data, a type of neural network model designed for NLP tasks called the Transformer architecture, and the need for fine-tuning. Broad applicability Unlike specialized deep learning models that were trained to address a specific medical issue or clinical need, LLMs possess versatile capabilities that span various domains, such as healthcare, finance, and education. As a result, a one-size-fits-all regulatory framework is ill-suited for LLMs, and oversight must be adaptable to address diverse industry-specific concerns. Real-time adaptation LLMs can adapt their responses in real-time, based on user input and evolving contexts. This dynamic behavior demands that regulatory oversight incorporates continuous monitoring and evaluation mechanisms to ensure responsible usage and adherence to ethical guidelines. This is similar to what adaptive AI-based medical technologies would require from regulators. Societal impact The widespread adoption of LLMs has the potential to fundamentally transform various aspects of society. Consequently, regulatory oversight must address not only the technical aspects of LLMs but also their broader ethical, social, and economic implications. Data privacy and security LLMs’ reliance on extensive training data raises concerns related to data privacy and security. Regulatory oversight should establish robust frameworks to protect sensitive information and prevent unauthorized access or misuse of these powerful models. These unique characteristics of LLMs necessitate a tailored approach to regulatory oversight. Such an approach must be adaptive, holistic, and cognizant of the diverse challenges and potential consequences that LLMs present, ensuring their responsible and ethical use across various domains. THE FDA’S PRE-LLM OVERSIGHT OF AI The United States’ Food And Drug Administration (FDA) has been leading the global discussions on regulatory oversight and has been a prominent example in providing regulations about emerging technologies from 3D printed medications to AI-based medical tools5. With the increasing adoption of digital health technologies, the FDA started regulating Software as a Medical Device (SaMD) that refers to software solutions that perform medical functions and are used in the prevention, diagnosis, treatment, or monitoring of various diseases or conditions. As a continuation of that approach, the FDA has been adapting its regulatory framework to specifically address AI and machine learning (ML) technologies in medical devices6. The FDA released a discussion paper that outlined their potential regulatory approach tailored to AI and ML technologies used in medical devices7. The discussion paper proposed a total product lifecycle (TPLC) approach to regulating AI/ML-based SaMD, which focuses on the continuous monitoring and improvement of these technologies throughout their lifespan. The proposed framework also emphasized the importance of transparency, real-world performance monitoring, and clear expectations for modifications and updates to AI/ML algorithms. Currently, the FDA does not have specific categories exclusively for AI-based technologies but evaluates them within

Read the complete text