Zebra-Llama: A Context-Aware Large Language Model for Democratizing Rare Disease Knowledge
Karthik Soman, Andrew Langdon, Catalina Villouta, Chinmay Agrawal, Lashaw Salta, Braian Peetoom, Gianmarco Bellucci, Orion J Buske · 2024
Read Zebra-Llama: A Context-Aware Large Language Model for Democratizing Rare Disease Knowledge by Karthik Soman, Andrew Langdon, Catalina Villouta, Chinmay Agrawal, Lashaw Salta, Braian Peetoom, Gianmarco Bellucci, Orion J Buske (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02657.txt.
Description
Abstract Rare diseases present unique challenges in healthcare, often suffering from delayed diagnosis and fragmented information landscapes. The scarcity of reliable knowledge in these conditions poses a distinct challenge for Large Language Models (LLMs) in supporting clinical management and delivering precise patient information—underscoring the need for focused training on these ’zebra’ cases.
Catalog record
| CREATOR | Karthik Soman, Andrew Langdon, Catalina Villouta, Chinmay Agrawal, Lashaw Salta, Braian Peetoom, Gianmarco Bellucci, Orion J Buske |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02657
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zebra-Llama: A Context-Aware Large Language Model for Democratizing Rare Disease Knowledge Corresponding author: [email protected] or [email protected] Karthik Soman, Andrew Langdon, Catalina Villouta, Chinmay Agrawal, Lashaw Salta, Braian Peetoom, Gianmarco Bellucci, Orion J Buske Abstract Rare diseases present unique challenges in healthcare, often suffering from delayed diagnosis and fragmented information landscapes. The scarcity of reliable knowledge in these conditions poses a distinct challenge for Large Language Models (LLMs) in supporting clinical management and delivering precise patient information—underscoring the need for focused training on these ’zebra’ cases. We present Zebra-Llama, a specialized context-aware language model with high precision Retrieval-Augmented Generation (RAG) capability, focusing on Ehlers-Danlos Syndrome (EDS) as our case study. EDS, affecting 1 in 5,000 individuals, exemplifies the complexities of rare diseases with its diverse symptoms, multiple subtypes, and evolving diagnostic criteria. By implementing a novel context-aware fine-tuning methodology trained on questions derived from medical literature, patient experiences, and clinical resources, along with expertly curated responses, Zebra-Llama demonstrates unprecedented capabilities in handling EDS-related queries. On a test set of real-world questions collected from EDS patients and clinicians, medical experts evaluated the responses generated by both models, revealing Zebra-Llama’s substantial improvements over base model (Llama-3.1-8B-Instruct) in thoroughness (77.5% vs. 70.1%), accuracy (83.0% vs. 78.8%), clarity (74.7% vs. 72.0%) and citation reliability (70.6% vs. 52.3%). Released as an open-source resource, Zebra-Llama not only provides more accessible and reliable EDS information but also establishes a framework for developing specialized AI solutions for other rare conditions. This work represents a crucial step towards democratizing expert-level knowledge in rare disease management, potentially transforming how healthcare providers and patients navigate the complex landscape of rare diseases. Model Access Model weights: https://huggingface.co/zebraLLAMA/zebra-Llama-v0.2 Custom built RAG API for rare diseases (focused on EDS): • Base URL: https://zebra-llama-rag.onrender.com • Endpoint: /search Jupyter Notebook Demo of Zebra-Llama: https://github.com/karthiksoman/zebra-Llama/blob/main/code/notebook/zebra_llama_v0.2_demo.ipynb Source code: https://github.com/karthiksoman/zebra-llama 1 Introduction The advent of Large Language Models (LLMs) has ushered in a new era of artificial intelligence, with these models demonstrating unprecedented capabilities in natural language understanding and generation [ 1 ] . Their potential applications span numerous fields, including healthcare, where they promise to revolutionize tasks ranging from information retrieval to clinical decision support. Recent efforts to adapt LLMs for medical applications broadly fall into three categories: specialized pre-training, domain-specific fine-tuning, and advanced prompting techniques. Notable examples include BioGPT, pre-trained on PubMed abstracts [ 2 ] , and MedAlpaca, fine-tuned on specialized medical datasets [ 3 ] . Advanced prompting strategies, exemplified by MedPaLM [ 4 ] and Medprompt [ 5 ] , have achieved results comparable to human experts on medical licensing exams. While these approaches have yielded promising results in various medical tasks, they often do not fill the gap for highly specialized medical domains, particularly in the context of rare diseases. Rare diseases present unique challenges in the medical field, affecting only a small portion of the population yet often eluding timely diagnosis and effective treatment [ 6 , 7 ] . The medical adage, ”When you hear hoofbeats, think horses, not zebras,” while pragmatic for common ailments, can inadvertently lead to overlooking rare conditions – the metaphorical ”zebras” of medicine [ 8 ] . This oversight can result in significant delays in diagnosis and treatment, profoundly impacting patient outcomes and quality of life. The application of LLMs to rare disease management presents a promising yet underexplored avenue. Unlike common medical conditions with abundant data, rare diseases suffer from a scarcity of information and expertise, making them particularly challenging for traditional LLM approaches. This scarcity underscores the need for specialized AI solutions that can effectively aggregate, contextualize, and disseminate the limited available information on rare diseases from various sources, potentially transforming the landscape of rare disease diagnosis and management. Ehlers-Danlos Syndrome (EDS) epitomizes these challenges, serving as a prime example of the ”zebras” in medicine that often go unrecognized. As a group of inherited connective tissue disorders, EDS manifests through a diverse array of symptoms, including joint hypermobility, skin hyperextensibility, and tissue fragility [ 9 ] . With a combined prevalence estimated at 1 in 5,000 individuals worldwide, EDS significantly impacts patient lives while frequently evading timely diagnosis [ 9 ] . The syndrome’s heterogeneous nature, comprising at least 13 subtypes with overlapping yet distinct clinical presentations, further complicates the diagnostic process [ 10 ] . This complexity, coupled with the syndrome’s relative rarity, often results in a prolonged diagnostic journey, misdiagnosis, and suboptimal management [ 11 ] . Moreover, the rapid evolution of EDS research, with emerging genetic markers and updated clinical criteria, creates an ever-expanding knowledge base that is challenging to navigate [ 12 , 13 , 14 ] . EDS thus presents an ideal case study for exploring how specialized AI solutions, particularly advanced LLMs, could revolutionize rare disease management by effectively aggregating, contextualizing, and disseminating the limited yet crucial information available from various sources. To address these challenges in EDS management and demonstrate the potential of specialized AI in rare disease contexts, we present Zebra-Llama, a novel context-aware language model tailored for EDS information management. Drawing from a diverse array of sources including medical literature, patient forums, and clinical resources, our model is designed to address the multifaceted nature of EDS concerns, ranging from clinical presentations to daily living challenges. Our approach significantly enhances the utility of Retrieval-Augmented Generation (RAG) in the EDS domain, as illustrated in Fig 1. While traditional base language models (Fig. 1A) may provide inaccurate or imprecisely cited information, and standard RAG implementations (Fig. 1B) often struggle with contextual relevance, Zebra-Llama (Fig. 1C) demonstrates superior performance in generating precise and accurately cited responses. This is achieved through our specialized context-aware fine-tuning methodology, which trains the model to effectively discern and utilize relevant information from retrieved contexts. Zebra-Llama not only improves the accuracy and relevance of generated answers but also enhances the model’s ability to provide proper citations, a critical feature for trustworthy medical information systems. By focusing on EDS, our model exemplifies how tailored AI solutions can potentially transform rare disease information management, offering a scalable approach to aggregating, contextualizing, and disseminating specialized medical knowledge. Our results demonstrate significant improvements in thoroughness, accuracy, clarity, and citation reliability compared to base models, paving the way for more accessible and reliable rare disease knowledge dissemination. This work represents a crucial step towards leveraging advanced AI to address the ”zebras” of medicine, potentially improving diagnosis, treatment, and overall care for individuals affected by EDS and, by extension, other rare conditions. Figure 1: Fig 1: Comparison of different approaches to EDS-related query handling. (A) Base Llama model generating answers without RAG context, resulting in potentially inaccurate information and hallucinated citations. (B) Base Llama model with RAG implementation, showing imprecise utilization of retrieved context and inclusion of irrelevant information and citations. (C) Zebra-Llama model demonstrating enhanced context-aware RAG capabilities, generating precise responses with accurate citations derived specifically from relevant portions of the retrieved context. The color-coding indicates the relevance of retrieved and generated information (green: relevant, red: non-relevant), highlighting Zebra-Llama’s improved ability to focus on pertinent information while generating responses. 2 Methods In developing Zebra-Llama, our specialized language model for Ehlers-Danlos Syndrome (EDS), we implemented a novel context-aware fine-tuning methodology designed to enhance the model’s performance in this specific medical domain. Our