Performance of a large language model in the informed consent process for participation in a clinical trial Authors: Rebeca Moscatel, Komal Aryal, David Chen, Allan S Detsky, Kieran L Quinn Published: 2026 ABSTRACT AbstractThe performance of large language models (LLMs) in the informed consent process for a randomized clinical trial (RCT) remains unknown. The objective of this proof-of-concept study was to evaluate whether a large language model constrained to trial-specific documentation can deliver technically accurate, reproducible responses to informed consent questions with minimal hallucination, as a foundational validation step prior to participant-facing evaluation; and to assess the feasibility of using a second large language model to reproduce document-grounded accuracy assessments in alignment with human evaluators, as a proof of concept for scalable evaluation, rather than to establish independent ground truth correctness. Mean accuracy scores across all responses were consistently rated high by humans (4.8, 95% CI: 4.7–4.9) and the moderator LLM (4.7, 4.6–4.8). Readability was appropriate (human: grade 7.5, 7.0–8.0; LLM: grade 6.4, 6.0–6.9) and there was near identical semantic consistency (human: 0.91, 0.89–0.92; LLM: 0.91, 0.89–0.92). Inter-rater reliability of accuracy ratings demonstrated significant differences between human assessors (κ = 0.1, 0.0–0.2) and the moderator LLM (κ = 0.8, 0.8–0.8). Human raters gave different accuracy ratings to the same answers generated by the consent LLM due to perceptions that responses were technically accurate but inadequate because they lacked conversational tone, awareness of context, and empathy. We developed a set of novel LLM-driven tools that improve the efficiency of the informed consent process by enabling people designing RCTs to answer potential subjects’ questions and validate those models efficiently with machines, rather than people.Subject terms: Clinical trials, Computer science Introduction Clinical trials are the backbone of medical innovation. Yet their staggering costs—averaging $19 million per trial1 – are driven mainly by the salaries of research personnel2. As trial designs and their measures become increasingly complex, the burden on research coordinators (RC) to support trial conduct intensifies, contributing to delays, increased variability in participant experience, and rising operational costs3. Improving the efficiency of trial conduct to accelerate the creation of new knowledge and provide better value to funders is therefore essential. Large language models (LLMs) are computational models capable of language generation and other natural language processing tasks that enable them to have conversational interactions with humans4. These emerging artificial intelligence (AI) tools offer a novel opportunity to improve trial efficiency by augmenting the traditional roles of human RCs in their conduct5. For example, several studies have demonstrated the efficiency of identifying eligible participants compared to manual review6,7, collecting data from consented participants throughout the trial8, and mimicking human-like qualities to establish trust and acceptance with participants9–11. However, their ability to accurately and reliably answer prospective participants’ questions about a randomized clinical trial (RCT) as part of the informed consent process—an ethically sensitive interaction—remains unknown. We developed a proof-of-concept LLM-based system to deliver and evaluate trial-specific consent information’, which we called the machine learning to optimize and accelerate efficiency in randomized clinical trials (MODEL-RCT), to support high-quality informed consent delivery12–14. To enhance the generalizability, reproducibility, and scalability of this novel tool, we developed a second moderator LLM to measure the performance of the consent LLM and ensure that rigorous quality and ethical standards are met12–14. Together, the two objectives of this study were to (1) measure the operating characteristics (accuracy, reproducibility, semantic consistency, and reading level) of the MODEL-RCT consent LLM’s responses to prospective trial participant questions using humans to evaluate its performance; and (2) to determine if a second moderator LLM can validate its performance, compared to human assessors as the reference standard. The informed consent process involves multiple dimensions. The current proof-of-concept study advances our work toward developing a fully functional trials consent intervention by intentionally focusing on evaluating technical accuracy and validating consistency within a constrained information environment as an early-stage, pre-participant-facing validation step. While readability was assessed to characterize linguistic accessibility, readability alone does not reflect user comprehension, and participant understanding cannot be inferred from these measures. Accordingly, this study evaluates information quality rather than participant-level outcomes such as understanding, perceived helpfulness, or adequacy of consent. Establishing reliable, document-constrained information delivery is a necessary prerequisite for any participant-facing application, as evaluation of user understanding, acceptability, or decision-making cannot be conducted meaningfully if the underlying information is not consistently accurate. By demonstrating that the system can provide reproducible, document-grounded responses under controlled conditions, this study provides the foundational evidence required to support subsequent phases of evaluation in real-world, participant-facing settings. This sequencing reflects the ethical importance of establishing reliable information delivery and control of potential LLM hallucinations before evaluating interactive, contextual, accessibility, and adequacy of consent. As such, the findings should be interpreted within this defined scope and not as an assessment of real-world consent interactions. Results Operating characteristics of MODEL-RCT The mean accuracy ratings across the three sets of 106 responses (n = 318) provided by the consent LLM were consistently rated high among all human assessors (Fig. 1 and Table 1). However, inter-rater reliability was slight (Table 1). Human assessors gave different accuracy ratings to the same answers generated by the consent LLM. Qualitative feedback from these assessors revealed that this arose due to perceptions that responses were inadequate, particularly when they lacked conversational tone, contextual awareness, or responsiveness to implicit concerns. These findings highlight that responses may be technically accurate while still being perceived as inadequate in supporting participant understanding. Analyses of the consent LLM’s semantic consistency by human assessors revealed that it generated near-identical meaning across all responses when answering the same question using different wording (Table 1). The responses provided by the consent LLM were found to be at a reading level equivalent to grade 7–8 (Table 1). Accuracy of the consent LLM responses to 106 trial-related questions about a real-world, ongoing RCT (NCT06792214), covering its purpose, risks, benefits, and participant voluntariness. The same consent LLM independently provided three responses for each of the 106 questions (LLM1, LLM 2, and LLM3). Each of the three sets of these 106 responses was then rated for accuracy by three independent human assessors (hollow shapes) and the same moderator LLM (solid shapes). The shapes and bars represent the mean accuracy rating and 95% confidence interval across each of the 106 responses provided by the consent LLM. The Y-axis is truncated at a lower limit of 4. Table 1Summary of informed consent performance metrics of the consent LLM responses to 106 trial-related questions about a real-world, ongoing RCT (NCT06792214), covering its purpose, risks, benefits, and participant voluntarinessPerformance metricAssessorInterpretationHumanModerator LLMAccuracy, mean (95% CI)4.8 (4.7–4.9)4.7 (4.6–4.8)High agreement (both)Inter-rater reliability, κ (95% CI)0.1 (0.0–0.2)0.8 (0.8–0.8)Slight agreement (AS-H); substantial agreement (AS-LLM)Semantic consistencya, mean (95% CI)0.91 (0.89–0.92)0.91 (0.89–0.92)Near identical alignment (both)Readability, mean grade (95% CI)7.5 (7.0–8.0)6.4 (6.0–6.9)Grade 6 level (AS-LLM)The same consent LLM independently provided three responses for each of the 106 questions. Each of the three sets of these 106 responses was then rated for accuracy by three independent human assessors and the same moderator, LLM.CI confidence interval.aMean semantic consistency across comparisons of the consent LLM responses (1 vs 2; 1 vs 3; and 2 vs 3). Summary of informed consent performance metrics of the consent LLM responses to 106 trial-related questions about a real-world, ongoing RCT (NCT06792214), covering its purpose, risks, benefits, and participant voluntariness The same consent LLM independently provided three responses for each of the 106 questions. Each of the three sets of these 106 responses was then rated for accuracy by three independent human assessors and the same moderator, LLM. CI confidence interval. aMean semantic consistency across comparisons of the consent LLM responses (1 vs 2; 1 vs 3; and 2 vs 3). The MODEL-RCT consent LLM deferred to the principal investigator in 4.4% of responses (n = 14 of 318). These deferrals represent intentional boundary behavior, occurring when responses required patient-specific interpretation or extended beyond document-grounded information. Such cases provide insight into model limitations and illustrate how the system manages uncertainty by avoiding unsupported inferences. This occurred when questions required patient-specific interpretation, including medication suitability or contraindications (e.g., drug–drug interactions, renal or hepatic impairment), changes in clinical status during the trial, individualized risk–benefit considerations, or safety decisions explicitly designated in the protocol to contact the treating physician or principal investigator. This behavior was consistent with the consent LLM’s prompt constraints to avoid inference and to redirect queries when document-grounded answers could not be provided, thereby mitigating the risk of generating inaccurate or unsupported guidance. The consent LLM demonstrated consistent constraint under adversarial conditions, meeting all prespecified safety criteria across repeated evaluations. Validation of the moderator LLM The mean accuracy ratings across the three sets of 106 responses (n = 318) provided by the consent LLM were consistently rated high by the moderator LLM, similar to the ratings provided by human assessors (Fig. 1 and Table 1). In contrast to the limited agreement observed across human assessors, the inter-rater reliability across the accuracy rating of the moderator LLM was substantial (Table 1). Analyses of the consent LLM’s semantic consistency by moderator LLM revealed that it generated near-identical meaning across all responses when answering the same question using different wording (Table 1). The responses provided by the consent LLM were assessed by the moderator LLM to be at a reading level equivalent to grade 6–7 (Table 1). Discussion This study demonstrates the proof-of-concept that constraining an LLM to trial-specific documentation can support accurate delivery of consent-related information and that such performance may be reliably evaluated using a self-validation approach. These findings establish foundational capabilities required for the safe development of AI-supported consent tools, representing a necessary step prior to participant-facing evaluation. Before such systems can be studied in real-world consent interactions, it is essential to establish that they can reliably deliver document-grounded information without inference or hallucination. This study addresses that prerequisite stage. Informed consent is a core ethical requirement of clinical research, grounded in the obligation to provide prospective participants with information that is accurate, consistent, and comprehensible. The increasing interest in using LLMs to support consent-related interactions introduces a distinct ethical risk: if such systems generate inaccurate, inconsistent, or hallucinated information, they may undermine participant understanding and compromise the integrity of the consent process. To ensure consistency throughout the study: first, we reported the operating characteristics (accuracy, reproducibility, semantic consistency, and reading level) of MODEL-RCT, an LLM designed to answer questions from prospective RCT participants like those of a human research coordinator during the enrollment process, using humans to evaluate its performance. Next, we investigated whether a second, independent LLM could validate the performance of the consent LLM and obviate the need for humans to perform this critical task. We found that the consent LLM accurately and reproducibly addressed more than 100 generated trial-related questions and did so at an appropriate reading level with nearly identical language and meaning. We also demonstrated the ability of a moderator LLM to validate the consent LLM’s performance. This novel tool will enable people designing RCTs to answer potential subjects’ questions and validate those models efficiently with machines, rather than people. Collectively, this innovative LLM-driven set of tools will rapidly enhance its generalizability and scalability across a range of future clinical trials, demonstrating the promise of AI in strengthening the conduct of clinical trials and improving the efficiency of research at scale6. The broad implementation of AI and multilingual LLMs in clinical trials may enhance overall value by accelerating recruitment and reducing costs, while maintaining high ethical standards. However, formal studies evaluating efficiency and cost-effectiveness are needed to substantiate these potential benefits. In the future, such tools may complement human RC by supporting specific information-delivery tasks. The present reliance on labor-intensive infrastructure also poses a barrier to equity and inclusion. However, the ability to deploy LLMs remotely and in multiple languages may broaden their reach and, therefore, the diversity of clinical trial participants. This is important because underrepresented populations remain excluded from many trials due to operational restrictions within traditional research workflows18. It may also enable participation by community hospitals that lack the rich trial infrastructure found in large academic centers. An important consideration raised by this study is the risk that AI-enabled consent tools, if developed without explicit attention to accessibility, could inadvertently reinforce existing digital and communication inequities. This proof-of-concept evaluation was intentionally limited to a text-based, non-interactive setting and therefore does not address the needs of individuals who require communication, cognitive, or multimodal support during consent discussions. While such tools may be well-suited to handling routine consent questions, their ethical deployment will require deliberate co-design with people with communication and cognitive disabilities, as well as integration with human RCs who can provide tailored support where needed. We identified the critical need to address potential accessibility adaptations and multimodal communication (including AAC), as prior work has shown that research processes relying on language-intensive interaction and standard consent procedures systematically exclude individuals with communication impairments such as aphasia, raising important ethical concerns related to equity, justice, and generalizability19. The low inter-rater reliability of the human RC’s accuracy ratings of the consent LLM responses was unexpected. However, the asynchronous design of our study may have played an important role. Our study did not involve live interactions between human participants and the consent LLM. Instead, the questions were initially generated by an LLM and validated by our research team. All 106 questions were then presented to the consent LLM as being posed by trial participants, and it was prompted to respond to all of them in its role as a research coordinator. Three independent human assessors then evaluated its responses. Human conversations are fundamentally interactive, not unilateral. Conversations involve a dynamic, synchronous exchange between participants, where ideas, intentions, emotions, and information are continuously negotiated in real time20,21. This process is co-constructed—meaning both parties contribute to the direction and substance of the dialogue22,23. Therefore, the lack of dynamic interaction, combined with the limited fidelity of the human-LLM, explains the differences in expectations regarding conversational adequacy rather than factual accuracy. This, in turn, may have reduced the perceived accuracy of responses based on human bias imparted by pre-conceived expectations of those responses. The lack of inter-rater agreement among human assessors highlights the need for caution when considering LLMs to augment, rather than replace, human RC in informed consent. At the same time, the high internal consistency of the document-constrained LLM raises a parallel question about variability in consent information delivery when multiple human coordinators are involved. Although overall accuracy ratings were high, these findings should be interpreted in the context of observed limitations in response adequacy and how well responses fully addressed the question. High accuracy does not necessarily imply that responses are sufficient to support fully informed decision-making. The identification of borderline responses and deferral behaviour provides important insight into model performance, suggesting that limitations arise primarily from responses that do not fully address the question rather than from factual errors. These findings reinforce the need for participant-facing evaluations to assess how such responses are interpreted and used in real-world consent interactions. To our knowledge, there are no empirical studies examining the consistency of consent responses across different RCs or within the same coordinator across participants. The strength of our study includes its novel application of an innovative technology, its rigorous evaluation of multiple elements germane to the consent process, and the development of a tool that can enhance the generalizability and scalability of the MODEL-RCT consent LLM through rapid validation of its performance in future trial applications. Our study’s limitations: (1) this study did not involve real-time participant interaction, assessment of consent adequacy or acceptability, or evaluation of accessibility adaptations. These elements were intentionally outside the scope of this early-stage validation study, which focused on establishing foundational performance characteristics prior to evaluation as a participant-facing intervention. As a result, findings should not be interpreted as evidence of effectiveness in participant-facing informed consent processes, but rather as evidence supporting progression to subsequent phases of evaluation; (2) we did not explicitly address issues of accessibility, digital exclusion, and co-design with participants with communication or cognitive needs, which bear further study prior to real-world deployment; (3) we also did not evaluate for bias in its responses. However, a recent trial demonstrated that no significant differences were observed in race, gender, or ethnicity between participants enrolled via AI-assisted versus manual screening, suggesting that generative AI did not introduce additional demographic bias; and (4) patient partners contributed to the early conceptualization of the research question but were not involved in model evaluation, data analysis, or interpretation of results for this proof-of-concept study. (5) An important limitation of this study is that it does not assess whether responses improve participant understanding or are perceived as helpful in supporting decision-making. While readability metrics such as Flesch–Kincaid provide an estimate of linguistic complexity, they do not capture comprehension or understanding as part of informed consent. Evaluation of these participant-level outcomes will require prospective, participant-facing studies incorporating measures of understanding, usability, and perceived helpfulness, which are the focus of ongoing and future work. (6) The use of an LLM as an evaluator introduces the potential for circularity, as both the generator and evaluator may share similar underlying representations and reasoning patterns. In this study, we did not use the moderator LLM to establish ground truth correctness, but rather to assess whether document-grounded accuracy evaluations could be consistently reproduced in alignment with human assessors, who served as the reference standard. While the high agreement observed suggests that LLM-based evaluation may offer a scalable approach to reproducing structured assessment tasks, it does not eliminate the need for human oversight, particularly in participant-facing or high-stakes applications. Future work should explore hybrid evaluation frameworks that combine human and automated assessment to mitigate potential bias. (7) We did not perform ablation studies, defined as the systematic removal or modification of individual system components to assess their contribution to overall performance, or other component-level analyses. While the observed high median accuracy supports overall system performance, it does not allow attribution of performance to specific components. In scenarios where performance may be suboptimal, the relative contribution of individual components cannot be determined from this study and warrants further investigation. (8) This study did not evaluate variability in performance across different categories of consent-related questions, such as risks, procedures, or eligibility criteria. It is possible that model performance varies across these domains, particularly in areas that require more nuanced interpretation or where responses must closely match what is being asked. (9) This study was based on a curated set of 106 questions derived from a single clinical trial, which may limit generalizability across different trials and consent contexts. Future work should evaluate performance across larger and more diverse datasets. We demonstrate that LLMs can play an important role in the informed consent process of a clinical trial by answering a validated list of potential questions of people considering enrollment. We also developed an LLM-based tool to validate the performance of our consent LLM in its application to future clinical trials to enhance its scalability and efficiency by enabling people designing novel RCTs to validate those models efficiently with machines, rather than people. The deployment of LLMs has the potential to transform clinical trials and pave the way for a future where AI-driven consent makes trials faster, fairer, more affordable, and accessible to all. Before LLMs can be considered for participant-facing use, it is therefore necessary to establish that they can reliably deliver trial-specific information without inference beyond source documents, and that their outputs can be evaluated in a consistent and reproducible manner. Future research should incorporate comprehensive co-design with patient and caregiver partners as a core design principle to enhance usability, acceptability, and ethical alignment of consent LLMs and ensure that AI-supported consent models expand access rather than narrow it. Methods This study is reported in accordance with the TRIPOD-LLM reporting guideline, as recommended by the EQUATOR Network, for studies developing, prompting, or evaluating LLMs. Although the present study does not involve individual prognosis or diagnosis, TRIPOD-LLM provides a structured framework to support transparent reporting of LLM architecture, prompting, data constraints, evaluation procedures, and reproducibility. Ethics approval was not required as the study did not involve participant-facing interaction and involved only secondary evaluation by human assessors. Study design and setting We conducted a proof-of-concept study of the MODEL-RCT consent LLM to evaluate whether a document-constrained LLM can accurately deliver trial-specific informed consent information in a controlled, non–participant-facing setting. This design was intentionally selected as an early-stage validation approach to assess core operating characteristics prior to real-world, participant-facing deployment. Our purpose at this point was not to compare performance across LLMs or against humans as the reference standard, but rather to measure the accuracy of a fully configured, document-constrained LLM system in responding to common questions about participation in a real-world clinical trial. In this design, prompt structure, document context, and generation parameters were treated as fixed components of the intervention, rather than variables for independent evaluation. First, we used an LLM (distinct from the consent LLM) to generate a set of 106 pre-scripted informed consent questions related to an ongoing RCT, the Paxlovid/Remdesivir Effectiveness for the prEvention of loNg coviD Clinical Trial (DEFEND; NCT06792214). These questions covered trial objectives, procedures, potential risks and benefits, and the voluntary nature of participation, reflecting the range of topics encountered in real-world consent discussions. However, the question set was designed to evaluate overall system performance and was not structured for stratified analysis by question category. Second, to mitigate bias and enhance the validity of these LLM-generated questions, three trained and experienced clinical trial RCs independently reviewed, refined, and validated them to ensure they reflected the questions commonly asked by real-world trial participants during consent discussions. This proof-of-concept study was intentionally conducted in a non-participant-interactive, text-only setting and did not include accessibility adaptations or co-design with individuals with communication or cognitive support needs. Third, we fed these questions into the consent LLM and prompted it to respond to them in the role of a research coordinator. To enable measurement of reproducibility, all questions were answered independently by the same consent LLM three times, producing three independent responses per question. Fourth, we evaluated the accuracy, reproducibility, semantic consistency, and reading level of the consent LLM’s responses using three humans as independent primary evaluators. We also studied the ability of a second moderator LLM to evaluate the performance of the consent LLM by assessing its agreement with human assessors, who served as the reference standard for document-grounded accuracy. The moderator LLM was not used to establish ground truth, but to evaluate whether accuracy assessments could be consistently reproduced in alignment with human judgment under the same document constraints. Each of the three sets of answers provided by the consent LLM was evaluated by three independent human assessors and by the moderator LLM three times. The consent and moderator LLMs were generative pre-trained transformer (GPT) 4, developed using OpenAI’s GPT-4o (version 1.2025.119) programmatic API. The study was conducted from November 7, 2024, to April 13, 2025. To assess falsifiability, we conducted a post-hoc adversarial question test designed to challenge the consent LLM outside the bounds of trial-specific information delivery. Twenty adversarial questions were generated using an LLM and grouped into five non–trial-related categories: general health, leisure activities, critical danger situations, emotional distress, and miscellaneous queries unrelated to the protocol. Each question was answered independently in triplicate by the consent LLM to assess response stability and replicability under adversarial conditions. All adversarial responses were independently reviewed by three experienced clinical trial RCs. Reviewers assessed whether responses met predefined validation criteria: (a) no provision of medical, psychological, or situational advice; (b) strict adherence to trial-documented information, with appropriate referral to the clinical or research team when information was unavailable; (c) correct activation and maintenance of the emergency protocol when applicable; and (d) absence of diagnostic statements or clinical interpretation. This testing framework was designed to evaluate whether the model would appropriately constrain its responses, refrain from inference, and reliably redirect queries beyond its intended scope. This post hoc study was conducted from January 20 to January 29, 2026. This study was designed as an initial proof-of-concept to evaluate the accuracy, reproducibility, and validation feasibility of a consent LLM constrained to trial documentation, prior to introducing live participant interaction or broader accessibility adaptations. Data and knowledge base The DEFEND trial’s study protocol, operations manual, and participant informed consent form were uploaded into the API and served as the knowledge base for both the consent and moderator LLM. The LLMs were tuned with a temperature of 0.2 to produce deterministic outputs. This low-temperature setting was intentionally selected to minimize response variability and reduce the risk of hallucination, prioritizing reproducibility and consistency in information delivery. This approach reflects a design choice aligned with potential real-world deployment in informed consent settings, where stability and reliability of responses are critical. Prompt engineering We developed MODEL-RCT as a consent LLM to function as a clinical trial research coordinator through iterative rounds of prompt engineering using the CREATE (character, request, examples, adjustments, type of output, extras) framework15. The consent LLM was prompted to provide a lay summary of the DEFEND trial objectives, procedures, risks, and benefits at a grade 6 reading level and support participant navigation of the consent process by answering all questions posed by the prospective participant. The consent LLM was further prompted to state that personal health information (PHI) should not be shared within the chatbot environment, owing to concerns over privacy, and that participants were consenting to the knowledge that any sharing of their PHI was considered non-secure. It was further instructed to invite participant questions, respond with factually accurate and accessible language, and guide the conversation until the participant affirmed consent readiness following a specific prompt. The consent LLM was told to redirect all unresolved clinical queries to the principal investigator (KLQ) to mitigate potential risks from providing inaccurate responses. We also prompted the consent LLM to provide links to the e-consent form and a post-consent survey to collect sociodemographic data to support electronic consent workflows in a separate, secure, and privacy-compliant environment. While the consent LLM was prompted to reference the availability of an electronic consent form and a post-consent survey, the study design deliberately constrained the interaction to a self-contained information-delivery phase, and no links were provided, nor were survey workflows activated. The moderator LLM’s internal decision-making process was intentionally constrained by prompt design to assess document-grounded technical accuracy only. Specifically, the moderator was instructed to compare each response directly against the same trial-specific source documents provided to the consent LLM and to refrain from evaluating conversational tone, emotional nuance, cultural sensitivity, or contextual adequacy. While this limited transparency into broader reasoning processes may constrain trust in participant-facing settings, it reflects a deliberate methodological choice aligned with the study’s proof-of-concept objective. Evaluation of emotional, cultural, and clarification-related dimensions of informed consent lies beyond the scope of this study and is being addressed in subsequent participant-facing investigations. The final prompt structure and knowledge base were finalized on April 18, 2025. To validate falsifiability, the moderator’s objective would be considered unsuccessful if the moderator LLM (i) produced varying ratings across repeated evaluations of identical responses, (ii) demonstrated poor agreement with human assessors, or (iii) failed to discriminate between accurate and inaccurate responses when constrained to the same trial documentation. Furthermore, based on findings from the initial adversarial testing, we implemented an additional constraint framework to strengthen safety and ethical guardrails. These constraints included: (1) temporal restrictions limiting responses to pre-trial consent information only; (2) an explicit prohibition on providing health advice or information outside the trial protocol; (3) clarification that the consent LLM does not function as a counselor or clinician; and (4) an emergency override protocol aligned with standard GPT emergency response guidance for situations involving potential imminent harm. These refinements were incorporated to ensure that the consent LLM consistently avoided inference, diagnostic reasoning, or advisory behavior when confronted with adversarial or high-risk queries. This final prompt adaptation and knowledge base were finalized on January 26, 2025. Outcomes The two primary outcomes were accuracy and reproducibility of the consent LLM responses. The secondary outcomes were the readability, semantic consistency, and perspectives of trained human RC on their accuracy ratings for the consent LLM. Readability was included to characterize linguistic accessibility of responses; however, no measures of participant comprehension, perceived helpfulness, or understanding were included, as the study was conducted in a non–participant-facing setting focused on technical performance. Response accuracy was measured using a 5-point Likert scale. We measured response readability using the Flesch-Kincaid Grade Level index (range 0–20; kindergarten to post-graduate) and the semantic consistency using Sentence-BERT to measure agreement in language and meaning by comparing cosine similarity between responses to the same question16. Readability was assessed using the Flesch–Kincaid Grade Level index, calculated through manual programming of Python to ensure consistent and reproducible scoring across all responses. Statistical analysis We used descriptive statistics to report all outcomes and Fleiss’ kappa (κ) to measure inter-rater agreement across human and moderator LLM accuracy assessments. Following observation of low inter-rater reliability among human assessors, we conducted a structured post-hoc debriefing discussion with the three clinical trial RC to explore factors contributing to variability in accuracy ratings. Insights from this discussion were synthesized using narrative synthesis to capture shared explanatory perspectives and to contextualize the quantitative findings, consistent with published guidance on the use of narrative synthesis for integrating qualitative insights alongside quantitative results17. All quantitative analyses were performed using GraphPad Prism version 10.4.2 and Python version 13.3.3. Supplementary information Supplements Acknowledgements The authors would like to express their gratitude to the patient partners who contributed to the initial conceptualization and framing of the research questions that informed this study. This study was supported by the Sinai Health Department of Medicine Research Fund. The analyses, conclusions, opinions, and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred. Author contributions K.L.Q. and R.M. conceptualized the study. K.L.Q., R.M., and D.C. conducted the analyses. K.L.Q. and R.M. verified the underlying data. K.L.Q., R.M., K.A., D.C., and A.S.D. wrote the manuscript. All the authors had access to the data in the study, contributed to the data interpretation and discussion, and approved the final manuscript version. K.L.Q. and R.M. had the final responsibility for the decision to submit the study for publication. Data availability The dataset from this study is held at Sinai Health. The full protocol, dataset, and prompt engineering strategy are available from the authors upon request. Code availability The consent and moderator LLMs were Generative Pre-trained Transformer (GPT) 4, developed using OpenAI’s GPT-4o (version 1.2025.119) programmatic API. The Python code used to calculate the Flesch–Kincaid Grade Level is available upon request (Supplementary Note 1). Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary information The online version contains supplementary material available at https://doi.org/10.1038/s41746-026-02745-9. References Associated Data Supplementary Materials Supplements Data Availability Statement The dataset from this study is held at Sinai Health. The full protocol, dataset, and prompt engineering strategy are available from the authors upon request. The consent and moderator LLMs were Generative Pre-trained Transformer (GPT) 4, developed using OpenAI’s GPT-4o (version 1.2025.119) programmatic API. The Python code used to calculate the Flesch–Kincaid Grade Level is available upon request (Supplementary Note 1).