Real-world validation of a multimodal LLM-powered pipeline for High-Accuracy Clinical Trial Patient Matching leveraging EHR data
Anatole Callies, Quentin Bodinier *, Philippe Ravaud *, Kourosh Davarpanah * · 2025
Read Real-world validation of a multimodal LLM-powered pipeline for High-Accuracy Clinical Trial Patient Matching leveraging EHR data by Anatole Callies, Quentin Bodinier *, Philippe Ravaud *, Kourosh Davarpanah * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15374.txt.
Introduction
Anatole Callies and three co-authors test a system that matches patients to clinical trials using health record files as they are. Trial sign up is slow because entry rules are complex and chart review takes much hand work. Text only models lose facts held in images and scans.
The new pipeline uses a reasoning language model that can read both text and images, plus a search step based on vector embeddings. On a public set of 288 patients with diabetes it reached 93 percent accuracy per entry rule. On real data from 485 patients across 30 sites and 36 trials it reached 87 percent. Gaps in records, not model skill, drove most of the misses.
Catalog record
| CREATOR | Anatole Callies, Quentin Bodinier *, Philippe Ravaud *, Kourosh Davarpanah * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15374
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Real-world validation of a multimodal LLM-powered pipeline for High-Accuracy Clinical Trial Patient Matching leveraging EHR data Anatole Callies, Quentin Bodinier *, Philippe Ravaud *, Kourosh Davarpanah * Abstract Background : Patient recruitment in clinical trials is hindered by complex eligibility criteria and labor-intensive chart reviews. Prior research using text-only models have struggled to address this problem in a reliable and scalable way due to (1) limited reasoning capabilities, (2) information loss from converting visual records to text, and (3) lack of a generic EHR integration to extract patient data. Methods : We introduce a broadly applicable, integration-free, LLM-powered pipeline that automates patient–trial matching using unprocessed documents extracted from EHRs. Our approach leverages (1) the new reasoning-LLM paradigm, enabling the assessment of even the most complex criteria, (2) visual capabilities of latest LLMs to interpret medical records without lossy image-to-text conversions, and (3) multimodal embeddings for efficient medical record search. The pipeline was validated on the n2c2 2018 cohort selection dataset (288 diabetic patients) and a real-world dataset composed of 485 patients from 30 different sites matched against 36 diverse trials. Results : On the n2c2 dataset, our method achieved a new state-of-the-art criterion-level accuracy of 93%. In real-world trials, the pipeline yielded an accuracy of 87%, undermined by the difficulty to replicate human decision-making when medical records lack sufficient information. Nevertheless, users were able to review overall eligibility in under 9 minutes per patient on average, representing an 80% improvement over traditional manual chart reviews. Conclusion : This pipeline demonstrates robust performance in clinical trial patient matching without requiring custom integration with site systems or trial-specific tailoring, thereby enabling scalable deployment across sites seeking to leverage AI for patient matching. Keywords— clinical trial patient matching, patient recruitment using AI, visual reasoning, multimodal embeddings, automated chart review 1 Introduction Patient recruitment remains a critical bottleneck in clinical development pipelines, with enrollment challenges alone accounting for 30% of phase III trial terminations and substantially prolonging drug development timelines ( Shah et al., 2021 ). At the core of this problem is the laborious task of identifying suitable patients — a step often complicated by increasingly stringent requirements for trial eligibility. Over the last two decades, the median number of eligibility criteria has surged by 58%, rising from 31 in 2001–2005 to 49 in 2016–2020 ( Shah et al., 2021 ). This added complexity is particularly prevalent in fields such as oncology, neurology, and infectious diseases, where the criteria counts can exceed 60 ( Shah et al., 2021 ), placing even greater demands on Clinical Research Coordinators (abbreviated as CRCs in the rest of the document). As a result, the pre-screening phase alone can consume up to 60% of total patient recruitment effort ( Penberthy et al., 2012 ). When done manually, each patient pre-screening takes on average 50 minutes in phase III trials. Because as many as 88% of them ultimately fail to meet the criteria, it takes on average more than seven hours to find one eligible patient for Phase III trials ( Penberthy et al., 2012 ). In the Related Work section, we review previous efforts to address this challenge, detailing their approaches and the performance they achieved. In the Methods section, we present our pipeline, and how it overcomes those limitations by leveraging several recent innovations. The Results section presents tangible evidence of our pipeline’s effectiveness, exceeding state-of-the-art performance on both a public benchmark (the n2c2 2018 cohort selection dataset) and our real-world dataset built from in-app user feedbacks. We also demonstrate how it significantly decreases pre-screening time. Finally, the Discussion section synthesizes our findings and suggests future enhancements around uncertainty calibration and patient-level recommendations. 2 Related work 2.1 Previous research Prior research has tackled this challenge through two main approaches. First, a ”patient-centric” approach to find the relevant trials for a given patient. And, on the contrary, a ”trial-centric” approach to find relevant patients for a given trial. In this study, we focus on the latter ”trial-centric” task and adapt previous approaches for a better real-world applicability. Many previous studies have proven the theoretical feasibility and effectiveness of LLM-based eligibility assessments. To start with, Wong et al., 2023 introduced a framework leveraging LLMs to convert eligibility criteria into queryable expressions and match them against electronic health records (EHRs). Their approach demonstrated a patient-level recall varying between 67.3% and 76.8%. Though, their focus was solely on the challenging area of oncology trials, making it difficult to generalize to other therapeutic areas. Similarly, Hamer et al., 2023 explored the use of LLMs, and particularly OpenAI’s InstructGPT (pre-ChatGPT chat assistant), to perform patient-trial matching on a synthetic patient dataset. They showed that while the AI standalone performance was sub-par (low criterion-level accuracy of 72% resulting in a patient-level recall of 50% only), it was still possible to use it as a copilot and thus achieve perfect patient-level recall and acceptable precision (71%) while reducing the number of criteria to be checked by experts by 90%. Then, Jin et al., 2024 introduced TrialGPT, a three-stage approach (retrieval, criterion-assessment, trial-level aggregation) focusing on the patient-centric task (i.e. not comparable to our trial-centric approach). The second stage, that is has in common with the trial-centric approach, leverages GPT-4 and achieves a 87% criterion-level accuracy on the SIGIR 2016 cohort ( Koopman & Zuccon, 2016 ). To evaluate the benefit of providing a rationale alongside the actual eligibility decision, Wornow et al., 2024 assessed the interpretability of LLM-generated assessments. Clinicians reviewed the rationales generated by the LLM for each eligibility decision, demonstrating that the model can produce coherent explanations for 97% of its correct decisions and even 75% of its incorrect ones. 2.2 Shortcomings Despite promising feasibility, above approaches fell short when applied in a real-world setting. This is mainly due to four challenges: 1. Lack of basic reasoning skills, limiting assessment quality : Despite the improvements made to chat models (a.k.a. single-turn models, i.e. models optimized for fast conversational interactions with humans as opposed to reasoning models optimized for problem solving), even the most advanced ones, like GPT-4o, fail at some simple tasks. For example, date-related calculations or assessing some logical expressions remains a challenge for them (e.g. Figure 1 shows GPT-4o struggling with simple date calculations or making a wrong decision on lab values check). Figure 1: Examples of GPT-4o struggling with basic evaluation 2. Reliance on expert-refined criteria, limiting broad applicability : Many of the aforementioned approaches ( Wornow et al., 2024 , Wong et al., 2023 ) rely on having experts ask the right questions or refine the eligibility criteria to make them less ambiguous. While it is interesting in the context of one particular trial, it is not compatible with a broadly applicable pre-screening tool. Having an LLM autonomously refine a criteria could also be an option, but as eligibility criteria typically undergo rigorous scrutiny to meet many constraints, modifying them could lead to unintended consequences, so we put aside this approach. 3. Lack of visual understanding, limiting medical record coverage : Furthermore, despite the rise of EHRs, numerous institutions still rely heavily on handwritten notes. Ekowo et al., 2023 showed that in a sample of 405 patient notes taken in 2020 and 2021 in the UK, 40% of notes