Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track Journal: Journal of Biomedical Informatics

Brian Ondov, William Xia, Kush Attal, Ishita Unde, Jerry He, Dina Demner-Fushman · 2025

Read Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track Journal: Journal of Biomedical Informatics by Brian Ondov, William Xia, Kush Attal, Ishita Unde, Jerry He, Dina Demner-Fushman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14096.txt.

Introduction

Brian Ondov and five co-authors report lessons from a shared task that rewrote biomedical abstracts in plain language, in a paper from 2025. Language models could open research to patients and caregivers, but errors here can harm, so strict testing is vital. The track ran at two major evaluation conferences.

Twelve teams from twelve countries entered systems from simple networks to large transformers. Human judges found top models matched human accuracy and completeness in full rewrites, but lagged in simplicity and brevity. Automatic scores correlated poorly with human judgment. Term replacement proved harder to spot than to fix once found.

Catalog record

CREATORBrian Ondov, William Xia, Kush Attal, Ishita Unde, Jerry He, Dina Demner-Fushman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-14096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track Journal: Journal of Biomedical Informatics Brian Ondov, William Xia, Kush Attal, Ishita Unde, Jerry He, Dina Demner-Fushman Abstract Objective : Recent advances in language models have shown potential to adapt professional-facing biomedical literature to plain language, making it accessible to patients and caregivers. However, their unpredictability, combined with the high potential for harm in this domain, means rigorous evaluation is necessary. Our goals with this track were to stimulate research and to provide high-quality evaluation of the most promising systems. Methods : We hosted the Plain Language Adaptation of Biomedical Abstracts (PLABA) track at the 2023 and 2024 Text Retrieval Conferences. Tasks included complete, sentence-level, rewriting of abstracts (Task 1) as well as identifying and replacing difficult terms (Task 2). For automatic evaluation of Task 1, we developed a four-fold set of professionally-written references. Submissions for both Tasks 1 and 2 were provided extensive manual evaluation from biomedical experts. Results : Twelve teams spanning twelve countries participated in the track, with models from multilayer perceptrons to large pretrained transformers. In manual judgments of Task 1, top-performing models rivaled human levels of factual accuracy and completeness, but not simplicity or brevity. Automatic, reference-based metrics generally did not correlate well with manual judgments. In Task 2, systems struggled with identifying difficult terms and classifying how to replace them. When generating replacements, however, LLM-based systems did well in manually judged accuracy, completeness, and simplicity, though not in brevity. Conclusion : The PLABA track showed promise for using Large Language Models to adapt biomedical literature for the general public, while also highlighting their deficiencies and the need for improved automatic benchmarking tools. † † graphicalabstract: 1 Introduction The inability of patients to understand available information about their health significantly impacts outcomes ( Stableford and Mettger 2007 ; Berkman et al. 2011 ; King 2010 ; Kindig et al. 2004 ) . While more information than ever is available on the web, the “language barrier” of professional-facing content, such as literature, can make it counterproductive ( Rosenberg et al. 2017 ; White and Horvitz 2009 ) . Though many consumer-facing knowledge bases exist, these are cumbersome and labor-intensive to update and thus typically do not include the latest medical knowledge from the literature. This is especially important for consumers seeking out the latest information about emerging health issues ( SeyyedHosseini et al. 2022 ) . The many recent successes of Artificial Intelligence suggest the possibility of automatically adapting text intended for a professional audience to a consumer audience. Though Text Simplification has been widely studied in the open domain ( Al-Thanyyan and Azmi 2021 ) , the unique lexicon of biomedical literature makes it a distinct sublanguage ( Friedman et al. 2002 ) . In this respect, the task overlaps with Machine Translation ( Stahlberg 2020 ) . Recent examples of applying Deep Learning to Text Simplification specifically in the biomedical domain include Van den Bercken et al. 2019 , Devaraj et al. 2021 , and Flores et al. 2023 . However, the existing body of research evaluates systems with disparate datasets that are mined from imperfect sources, making progress difficult to measure. With the aims of providing common benchmarks, encouraging progress, and rigorous expert evaluation, we hosted the Plain Language Adaptation of Biomedical Abstracts (PLABA) shared task at the Text REtrieval Conference (TREC), spanning 2023 and 2024. This track included tasks for rewriting abstracts from the biomedical literature for the general public, as well as identifying expert terms and providing replacements or additional text for clarification. By encouraging creation of systems that can coherently rewrite entire abstracts, but at the sentence level, we hope to allow the public to engage with the latest research, while providing fine-grained provenance of each statement to ensure transparency and reliability. Across both years, twelve teams spanning twelve countries participated, submitting a total of 38 runs. Systems were diverse and powered by a variety of models, from Multilayer Perceptrons, to encoder-only transformer models such as BioBERT ( Lee et al. 2020 ) and RoBERTa ( Liu et al. 2019 ) , to encoder-decoder models like BART ( Lewis et al. 2020a ; Yuan et al. 2022 ) and T5 ( Raffel et al. 2020 ) , to modern, instruction-tuned decoder-only Large Language Models, including variants of Llama ( Touvron et al. 2023 ) , GPT ( Brown et al. 2020 ; Ouyang et al. 2022 ) , Gemini ( Team et al. 2023 ) , and Mistral ( Karamcheti et al. 2021 ) . We find that the best performing systems can be highly factually accurate while making abstracts much more comprehensible to the general public. Still, occasional falsehoods and hallucinations mean that care must be taken in deploying such systems, especially given the risk of harm in the biomedical domain. Additionally, the task revealed there is work to be done in automatically evaluating both the simplicity and factuality of outputs. We hope the lessons from the PLABA shared task will inform future tasks and research directions. Statement of Significance Problem Patients and caretakers cannot keep up with biomedical research due to technical language and jargon. What is Already Known Advanced Artificial Intelligence systems have sought to simplify professional text for a general audience. What This Paper Adds We stimulate methods research by providing a shared task and provide extensive manual review of outputs by biomedical experts to assess accuracy. Who would benefit from the knowledge in this paper Researchers using Artificial Intelligence to generate public-facing biomedical text output; clinicians who need to explain current research to patients. 2 Related Work Shared tasks on text simplification go back at least to Task 1 of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) ( Specia et al. 2012 ) . This task challenged teams to rank replacement words by simplicity, given context for disambiguation. Lists of potential lexical replacements for this task came from SemEval 2007 Task 10, for which the goal was to provide substitutions, regardless of simplicity ( McCarthy and Navigli 2007 ) . Both these tasks, however, were open domain, and did not focus on scientific text, which adds the complexity of technical language or jargon. Beginning with a pilot in 2021, the SimpleText track ( Ermakova et al. 2021 ; Ermakova et al. 2022c ; Ermakova et al. 2023c ; Ermakova et al. 2024b ) has been hosted yearly at the Conference and Labs of the Evaluation Forum (CLEF), a long-running European initiative with similar goals to TREC. As with PLABA, the SimpleText track aims to evaluate systems for simplifying scientific text. However, SimpleText has largely focused on Computer Science literature, with two relevant exceptions. In 2022, SimpleText data included a narrowly focused biomedical sub-corpus, comprising articles resulting from searching PubMed and Google Scholar for muscle health and hypertrophy. In 2024 a similar sub-corpus was included but expanding the search to articles on “health and medicine.” Each offering of SimpleText has had variations of three core tasks. Task 1 involves identifying passages of a scientific article that would be helpful for understanding by the general public ( SanJuan et al. 2022 ; SanJuan et al. 2023 ; SanJuan et al. 2024 ) .

Read the complete text