Hierarchical prompting with reasoning large language models for immune checkpoint inhibitor-associated myocarditis evidence extraction pilot study
Authors: Enshuo Hsu, Sheng-Chieh Lu, Sara Ebrahimi, Noah Beinart, Keila C Ostos-Mendoza, Anita Deswal, Kirk Roberts, Trey Kell, Jeff Jin, Nicolas L Palaskas
Published: 2026
ABSTRACT
AbstractAimsImmunotherapy with immune checkpoint inhibitors (ICIs) is an effective treatment for many cancers, but it can induce immune-related adverse events (irAEs). Among those, myocarditis is a severe complication associated with high morbidity and mortality. To support the rapidly emerging research, labour-intensive manual data extraction is often needed. Recent studies adopted large language models (LLMs) to systematically process clinical notes and identify patients with irAEs. However, research gaps persist: (i) For ICI-associated myocarditis, the applicability and scalability of LLMs have not been systematically evaluated; (ii) most existing studies focused on patient-level irAE detection while omitting the clinical nuances; and (iii) cutting-edge LLM techniques have not been adopted. Our aim was to evaluate an LLM approach to extract ICI-associated myocarditis evidence, including inflammatory infiltrate, life-threatening arrhythmias, heart failure, and stroke, from clinical notes.Methods and resultsWe collected clinical notes of patients with ICI-associated myocarditis at a single centre to develop and evaluate LLM-based methods that convert free-text clinical notes into structured data with entities (e.g. diagnosis, treatment, and imaging) and attributes (e.g. date, assertion, and status). We systematically evaluated three techniques: LLM reasoning, context engineering, and hierarchical prompting against ground truth created by a cardio-oncologist. Our proposed sentence-based context engineering and hierarchical prompting method, with a reasoning LLM, is significantly more accurate than the research trainees (F1 score of 0.6332 vs. 0.4933, P < 0.001), faster (37 min vs. 72 h per 1000 notes), and more cost-effective ($2.04 USD per 1000 notes).ConclusionWe proposed an optimized method that integrates hierarchical prompting, context engineering, and a reasoning LLM for ICI-associated myocarditis evidence extraction to support physicians and researchers in rapid clinical data collection for research to make diagnostic and treatment inferences.Keywords: Natural language processing, Large language models, Prompt engineering, Immune checkpoint inhibitor, Myocarditis
Introduction
Immunotherapy with immune checkpoint inhibitors (ICIs) is one of the latest pillars being added to several cancer treatment regimens. Since their introduction, a rapid expansion of ICI clinical trials and the approval of new agents for various indications has followed, and nine agents have been approved for the management of at least sixteen cancer groups. A striking increase in eligible patients receiving ICIs went from 1.54% in 2011 to 56.55% in 2023.1 While most agents target the classic checkpoints cytotoxic T lymphocyte associated protein-4 (CTLA-4) and programmed cell death protein-1/programmed death ligand-1 (PD-1/PD-L1), new agents targeting other checkpoints are currently being tested in clinical trials and will continue to expand the scope of immunotherapy. Despite the improvements in cancer outcomes, ICIs have been associated with a myriad of toxicities, referred to as immune-related adverse events (irAEs). Although the majority of these toxicities are low grade, the burden of affected patients can reach up to 50%.2 Heterogeneity in the diagnosis and management of irAEs introduces a challenge in understanding their true incidence, morbidity, and mortality. Cardiovascular irAEs, particularly myocarditis, are associated with higher morbidity and mortality compared with other irAEs.3 The field of cardio-oncology requires rapid development of research to address, diagnose, and treat ICI-associated myocarditis. In the current research practice, manual data extraction and clinical note interpretation are often required. Manually converting unstructured clinical note data to structured data elements is not only time-consuming and expensive but also prone to error.
Fortunately, recent natural language processing (NLP) advances have made systematic processing of unstructured narrative clinical data feasible for research and care improvement.4–6 Furthermore, large language models (LLMs), such as GPT, enable scalable extraction of diverse data elements from clinical notes, generating structured outputs of quality comparable to, or surpassing, that of human experts, while markedly reducing the cost and time required for manual annotation, thereby expediting downstream analyses.7,8 In oncology, previous studies have examined the performance of LLMs in cancer progression projection and capturing tumour characteristics and cancer diagnosis information from progress, histopathology, and imaging notes.9–12 Moreover, researchers have applied LLMs to enhance the detection of irAEs and concluded that GPT-like models are valuable tools for accurately identifying patients who develop irAEs.13–15
Despite the exciting advancements, several research gaps persist: first, the applicability and scalability of adapting LLMs for ICI-associated myocarditis evidence extraction in the cancer population have not been systematically evaluated. The accuracy and cost-efficiency of implementing LLM-based NLP information extraction pipelines in this domain remain unknown. Secondly, most existing studies on irAEs detection focus on flagging patients (i.e. cohort identification) while omitting note-level and mention-level nuances. This study presents methods that capture the granular data points as structured entities and attributes, and ultimately constructs a dataset that can be queried to support evidence-based medicine. Finally, we evaluated cutting-edge techniques in LLM prompting and reasoning, and proposed a high-performing, efficient, and low-cost system that can be generalized to other medical domains.
Methods
Data source and annotation
At the University of Texas MD Anderson Cancer Center in Houston, TX, USA, we have a cohort of 103 patients who presented to MD Anderson Cancer Center with suspected myocarditis on dates ranging from 2013 to 2019 and were diagnosed with ICI-associated myocarditis, meeting possible, probable, or definite myocarditis per Bonaca adjudication criteria. Diagnostic tests performed included measurement of cardiac biomarkers, electrocardiogram, echocardiography, and select patients had advanced testing, including fluorodeoxyglucose positron emission tomography computed tomography, cardiac magnetic resonance, and endomyocardial biopsy. We randomly sampled 10 patients from a larger cohort of patients who had endomyocardial biopsy results showing varying degrees of inflammatory infiltrate and myocardial injury for the diagnosis of myocarditis. These patients were also diagnosed retrospectively using International Cardio-Oncology Society diagnostic criteria16 and met definite category upon applying Bonaca adjudication criteria.17 We collected their clinical notes within 90 days before and after their initial ICI dose (all 10 patients sampled developed myocarditis prior to 90 days after ICI initiation). A total of 2258 unique clinical notes were collected. As a pilot effort, a random subset of 50 clinical notes was manually annotated as the ground truth. Of these, five notes were randomly chosen as the development set for prompt engineering and model selection, and 45 notes were used as the test set for evaluation. Note that the development and test set split was at the note level instead of the patient level. The development set was used for few-shot examples in the prompt templates. Sample size calculation is detailed in the Supplementary Implementation Details – Statistical analysis – Sample Size Determination section. We developed a comprehensive annotation guideline, detailing the entity classes, entity types, and attributes with definitions and examples. The guideline is used to develop (i) the annotation tool for human annotators and (ii) the prompts for LLM-based information extraction. The annotation guideline is available in the Supplementary material online, Table S1. Our manual annotation comprises two phases: two cardio-oncology research trainees (S.E. and N.B.) conducted an initial comprehensive note review independently and annotated the myocarditis-related entity spans (e.g. symptoms, treatments, and diagnosis of myocarditis) and attributes (e.g. occurrence date). Inter-annotator agreement was evaluated using token-level Cohen’s kappa and pairwise F-measure.18,19 A senior physician (N.L.P.) then conducted a final review to resolve the discrepancies and finalize the ground truth. An open-source graphical interface tool, Label Studio,20 was used for the manual annotation. The annotation schema follows the annotation guideline. In both phases, the annotators use the annotation guideline as the source of truth. This study was approved by the University of Texas MD Anderson Cancer Center Institutional Review Board (IRB #2021-0100).
Methodology overview
Our information extraction system inputs clinical notes, extracts structured entities and attributes, and constructs datasets that can be queried for patients with myocarditis (Figure 1A). We propose divide-and-conquer prompting strategies, including context engineering and prompt engineering. The context engineering strategy divides a long clinical note into manageable segments. Clinical notes are chunked into (i) paragraphs or (ii) sentences before being fed into LLM agents for entity and attribute extraction (AE). The hierarchical prompt engineering strategy divides the comprehensive information extraction guideline into multiple logical workflows, which are then executed by a series of LLM agents (Figure 1B). Entity and attribute data extraction performance will be compared between LLMs and cardio-oncology research trainees with the ground truth annotated by an experienced cardio-oncologist (N.L.P.). Specifics on the LLM method development are discussed below, involving: (i) reasoning vs. non-reasoning LLMs, (ii) whole document vs. paragraph and sentence contextualization (i.e. context engineering), and (iii) one-step vs. hierarchical prompting.
Central illustration. (A) Objective: utilize both context engineering and hierarchical prompting to extract structured myocarditis-related entities and attributes from free-text clinical notes, and subsequently construct structured datasets for patients with myocarditis. (B) Methods: context engineering was performed by chunking a clinical note into manageable text units before feeding it into large language model agents. Our hierarchical prompting method is comprised of eleven large language model agents: three entity extraction agents (‘Entity extractors’) and eight AE agents (‘Attribute extractors’) for granular information extraction. (C) Results: our proposed method achieved a significantly improved entity and AE F1 score over cardio-oncology research trainees, while reducing the data processing time from 72 h of manual labour to under 1 h of computer time per 1000 clinical notes. MRI, magnetic resonance imaging.
Information extraction schema for myocarditis evidence
The diagnosis of ICI-associated myocarditis is complex, comprising diagnostic criteria from the clinical signs and symptoms, cardiac biomarkers, electrocardiograms, echocardiograms, cardiac magnetic resonance imaging, and endomyocardial biopsy. The variables extracted focused on variables that typically do not exist in a structured format in the electronic medical record, such as symptoms, as opposed to laboratory values that can be easily exported in a structured format. To effectively capture relevant variables, we designed a structured information extraction schema, inspired by the named entity recognition (NER) and AE tasks in NLP, that is comprised of entities and attributes. An entity is a noun phrase of interest from a clinical note. For example, in the note text, ‘Pt admitted from EC with 1 week h/o low grade fever and rash and cough for 2–3 weeks,’ the terms ‘fever’ and ‘cough’ are entities. We classified entities into eight classes (‘entity classes’): signs and symptoms, diagnosis, outcome, electrocardiogram (EKG)/telemetry findings, echocardiogram (ECHO), cardiac magnetic resonance imaging (MRI), procedure, and treatment. For example, the entity ‘fever’ has ‘signs and symptoms’ as the class. An attribute is a normalized value assigned to an entity. For example, a fever that occurred on 10 August 2025 has a ‘date occurred’ attribute with a value of ‘08/10/2025’. We defined three universal attribute types for all entity classes and several other attribute types for specific entity classes. The universal attribute types include: (i) Entity type, which is the subtype nested under the entity classes (e.g. ‘fever’ and ‘dyspnoea’ under the ‘signs and symptoms’ class; ‘myocarditis’ and ‘myocardial infarction’ under the ‘diagnosis’ class; ‘steroids’ under the ‘treatment’ class). (ii) Assertion, which is the presence of the entity, assigned as one of ‘presence’, ‘absence’, or ‘possible’; (iii) date occurred, which is the date when the entity occurred in MM/DD/YYYY format. Other attribute types include ‘left ventricular ejection fraction’ value for the ‘cardiac MRI’ class, ‘EMB result’ for ‘procedure’ class, and ‘date intervention started’ for ‘treatment’ class. An example clinical note with expected structured outputs is shown in Table 1 and in Supplementary material online, Table S2, with more details. The full list of entity classes and attributes is available in Table 2.
Table 1Example of a clinical note and expected structured outputsClinical noteExpected outputs ‘… History of Present Illness: Patient presented to the emergency center 8/10/25 for evaluation of shortness of breath and restlessness for the last few days. Further evaluation showed patient in complete heart block with a bradycardic rate (HR 27 BP 80/50 s), therefore cardiology placed transvenous pacer in the EC and he was subsequently transferred to MICU for closer monitoring…’ [{ “entity_text": “shortness of breath”, “entity_class": “Signs and symptoms”, “attr": { “entity_type": “Dyspnea”, “date_occurred": “08/10/2025”, “assertion": “Presence”}},{ “entity_text": “complete heart block with a bradycardic rate”, “entity_class": “EKG/Telemetry findings”, “attr": { “entity_type": “Complete AV block”, “date_occurred": “08/10/2025”, “assertion": “Presence”}},{ “entity_text": “transvenous pacer”, “entity_class": “Treatment”, “attr": { “entity_type": “Transvenous temporary pacer”, “assertion": “Presence”}},{ “entity_text": “MICU”, “entity_class": “Treatment”, “attr": { “entity_type": “Upgrade level of care”, “assertion": “Presence” }}]A simplified structured output is presented in this table. The raw LLM outputs and complete JSON schema after post-processing are available in the Supplementary material online, Table S2.EC, emergency centre; MICU, medical intensive care unit; AV, atrioventricular; LLM, large language model; JSON, JavaScript Object Notation.
Example of a clinical note and expected structured outputs
A simplified structured output is presented in this table. The raw LLM outputs and complete JSON schema after post-processing are available in the Supplementary material online, Table S2.
EC, emergency centre; MICU, medical intensive care unit; AV, atrioventricular; LLM, large language model; JSON, JavaScript Object Notation.
Table 2Entities in the manually annotated ground truthMyocarditis entity extractionEntity classEntity typeTotal (n = 50)Test set (n = 45)Cardiac MRINot applicable84DiagnosisMyocarditis3832Congestive heart failure64Life-threatening arrhythmia22Venous thrombosis22Myocardial infarction11ECHOPericardial effusion44EKG/Telemetry findingsAtrial fibrillation or flutter169Complete AV block1010Sinus rhythm109Right bundle branch block33Non-sustained ventricular tachycardia22OutcomeLeft ventricular ejection fraction107Cardiotoxicity outcome11conduction disorder11ProcedureEMB1614LHC1010Signs and symptomsDyspnoea5248Fever5031Cough1711Lower limb oedema1612Fatigue1515Chest pain1211Palpitations63Syncope32Presyncope11Weight gain11TreatmentSteroids6254Oxygen therapy3428Presentation management2821Upgrade level of care2121Invasive ventilation1917Non-invasive ventilation1818Permanent pacemaker1111Pharmacologic cardioversion82Plasmapheresis88Transvenous temporary pacer88Conventional DMARDs21IL-6 inhibitor22TNF-a inhibitors22Anti-B-cell therapy11Vasopressors11 Total 538445Myocarditis attribute extractionAttributeValueTotalTest setAssertionPresence405332Absence9473Possible97Date occurred6037Date reported2016
Entities in the manually annotated ground truth
Development of generative information extraction methods with large language models
The use of LLMs for clinical information extraction is an established method featuring high performance, low labour, and flexibility. The workflow in general involves two steps: (i) prompting LLMs to review clinical text and generate semi-structured outputs and (ii) post-processing the outputs to create structured data.21 In this section, we describe our evaluation of the latest advancements in the LLM and NLP field: reasoning LLMs, context engineering, and hierarchical prompting for myocarditis entities and attributes extraction.
Reasoning LLMs are post-trained LLMs that perform chain-of-thought reasoning before generating responses. When prompted, they start with generating reasoning (‘thinking’) tokens enclosed between a pair of XML tags (e.g. ‘…’). For example,‘ Okay, let's look at the user's query. They provided a medical note snippet and want me to extract specific entities related to myocarditis due to ICI toxicity…’After the reasoning, a response is generated after the ending XML tag:‘ Okay, let's look at the user's query … There is no myocarditis mentioned, therefore the output should be an empty list []’Note that both the reasoning and response steps are automatically regulated by the LLM within the same output. Our post-processing logic parses results from the final responses. In this study, we adopted a lightweight yet high-performing reasoning LLM, GPT-OSS (‘GPT-OSS-120B’).22 As a comparison, we also evaluated a baseline non-reasoning LLM, Llama4-Scout (‘Llama-4-Scout-17B-16E-Instruct’)23 and a medical non-reasoning LLM, Medgemma (‘medgemma-27b-text-it’).24
Reasoning LLMs are post-trained LLMs that perform chain-of-thought reasoning before generating responses. When prompted, they start with generating reasoning (‘thinking’) tokens enclosed between a pair of XML tags (e.g. ‘…’). For example,
‘ Okay, let's look at the user's query. They provided a medical note snippet and want me to extract specific entities related to myocarditis due to ICI toxicity…’After the reasoning, a response is generated after the ending XML tag:
‘ Okay, let's look at the user's query. They provided a medical note snippet and want me to extract specific entities related to myocarditis due to ICI toxicity…’
After the reasoning, a response is generated after the ending XML tag:
‘ Okay, let's look at the user's query … There is no myocarditis mentioned, therefore the output should be an empty list []’Note that both the reasoning and response steps are automatically regulated by the LLM within the same output. Our post-processing logic parses results from the final responses. In this study, we adopted a lightweight yet high-performing reasoning LLM, GPT-OSS (‘GPT-OSS-120B’).22 As a comparison, we also evaluated a baseline non-reasoning LLM, Llama4-Scout (‘Llama-4-Scout-17B-16E-Instruct’)23 and a medical non-reasoning LLM, Medgemma (‘medgemma-27b-text-it’).24
‘ Okay, let's look at the user's query … There is no myocarditis mentioned, therefore the output should be an empty list []’
Note that both the reasoning and response steps are automatically regulated by the LLM within the same output. Our post-processing logic parses results from the final responses. In this study, we adopted a lightweight yet high-performing reasoning LLM, GPT-OSS (‘GPT-OSS-120B’).22 As a comparison, we also evaluated a baseline non-reasoning LLM, Llama4-Scout (‘Llama-4-Scout-17B-16E-Instruct’)23 and a medical non-reasoning LLM, Medgemma (‘medgemma-27b-text-it’).24
Context engineering focuses on prompting LLMs with a concise context that is helpful for making the correct responses. Clinical notes are often long and content-rich, resulting in the needle-in-the-haystack challenge, which causes LLMs to fail to retrieve important information. Following our recent work,25 we adopted two context engineering methods: (a) paragraph-based, where a clinical note is segmented into paragraphs by blank lines (i.e. ‘n\n’). We prompt LLMs with a paragraph at a time; (b) sentence-based, where a clinical note is first segmented into sentences. A sliding window of two sentences, before and after the unit, is collected as context. We prompt LLMs with a sentence-context pair at a time. Therefore, instead of processing the entire clinical note, LLMs only process a sentence at a time. The technical details of our unit-context algorithm are described in the Supplementary Implementation Details section. As a comparison, we also evaluated the baseline method (‘document-as-context’), in which the LLMs are prompted to process the entire clinical note in a single context window. We conducted a sensitivity analysis comparing different unit-context methods and found that the paragraph-based method achieved the best performance for most LLMs, but the sentence unit with sliding window method delivered optimal performance with GPT-OSS combined with hierarchical prompting (Supplementary material online, Table S3).
Context engineering focuses on prompting LLMs with a concise context that is helpful for making the correct responses. Clinical notes are often long and content-rich, resulting in the needle-in-the-haystack challenge, which causes LLMs to fail to retrieve important information. Following our recent work,25 we adopted two context engineering methods: (a) paragraph-based, where a clinical note is segmented into paragraphs by blank lines (i.e. ‘n\n’). We prompt LLMs with a paragraph at a time; (b) sentence-based, where a clinical note is first segmented into sentences. A sliding window of two sentences, before and after the unit, is collected as context. We prompt LLMs with a sentence-context pair at a time. Therefore, instead of processing the entire clinical note, LLMs only process a sentence at a time. The technical details of our unit-context algorithm are described in the Supplementary Implementation Details section. As a comparison, we also evaluated the baseline method (‘document-as-context’), in which the LLMs are prompted to process the entire clinical note in a single context window. We conducted a sensitivity analysis comparing different unit-context methods and found that the paragraph-based method achieved the best performance for most LLMs, but the sentence unit with sliding window method delivered optimal performance with GPT-OSS combined with hierarchical prompting (Supplementary material online, Table S3).
Hierarchical prompting is a practical strategy that divides a complex task into multiple small-scale subtasks that LLMs are capable of. Each subtask is executed by an LLM instance (‘agent’) with a specific goal assigned in user prompts. Considering our extraction schema, we designed a workflow comprised of three entity-extraction agents and eight subsequent attribute-extraction agents, as illustrated in the Supplementary material online, Figure S1. The entity-extraction agents are: (a) the sign-diagnosis-outcome agent that extracts entity classes and types for ‘signs and symptoms’, ‘diagnosis’, and ‘outcome’ entities. (b) The imaging agent that extracts entity classes and types for ‘EKG/telemetry findings’, ‘ECHO’, and ‘Cardiac MRI’ entities. (c) The treatment-procedure agent that extracts entity classes and types for ‘treatment’ and ‘procedure’ entities. We assigned conceptually related entity classes to the same agents. Sentences and contexts described in the previous section are fed into all three agents for entity extraction. Once an entity is extracted and assigned an entity class, a corresponding AE agent will review its context (i.e. a window of 4096 characters before and after) and generate attributes. For example, an entity with class ‘diagnosis’ is reviewed by the diagnosis AE agent for attributes: ‘assertion’ and ‘date occurred’. The user prompts of each agent are available in the Supplementary material online, Table S4. As a comparison, we evaluated the baseline method that prompts LLMs with a single prompt (‘one-step’ method) (see Supplementary material online, Table S5).
Hierarchical prompting is a practical strategy that divides a complex task into multiple small-scale subtasks that LLMs are capable of. Each subtask is executed by an LLM instance (‘agent’) with a specific goal assigned in user prompts. Considering our extraction schema, we designed a workflow comprised of three entity-extraction agents and eight subsequent attribute-extraction agents, as illustrated in the Supplementary material online, Figure S1. The entity-extraction agents are: (a) the sign-diagnosis-outcome agent that extracts entity classes and types for ‘signs and symptoms’, ‘diagnosis’, and ‘outcome’ entities. (b) The imaging agent that extracts entity classes and types for ‘EKG/telemetry findings’, ‘ECHO’, and ‘Cardiac MRI’ entities. (c) The treatment-procedure agent that extracts entity classes and types for ‘treatment’ and ‘procedure’ entities. We assigned conceptually related entity classes to the same agents. Sentences and contexts described in the previous section are fed into all three agents for entity extraction. Once an entity is extracted and assigned an entity class, a corresponding AE agent will review its context (i.e. a window of 4096 characters before and after) and generate attributes. For example, an entity with class ‘diagnosis’ is reviewed by the diagnosis AE agent for attributes: ‘assertion’ and ‘date occurred’. The user prompts of each agent are available in the Supplementary material online, Table S4. As a comparison, we evaluated the baseline method that prompts LLMs with a single prompt (‘one-step’ method) (see Supplementary material online, Table S5).
Evaluation
We evaluate the performance of (i) entity extraction and (ii) entity and attribute end-to-end extraction with precision, recall, and F1. Formulas and details are available in the Supplementary Implementation Details section. For entity extraction evaluation, we define the true positive (TP), false positive (FP), and false negative (FN) following common definitions in clinical NER studies.26 The LLM-extracted entities are considered TP if the span overlaps with a ground truth entity of the same entity class; FP if an extracted entity does not match a gold entity with the same class; FN if a gold entity does not match an extracted entity of the same class. For entity and attribute end-to-end evaluation, we define the TP as attributes having matched entity spans, entity types, and attribute values; FP as among the LLM-extracted entities, having matched entity spans but failed to match the attribute, or failed to match span with any ground truth entities; and FN as among ground truth entities, span matches an LLM-extracted entity but attribute does not match, or span does not match any LLM-extracted entities. We then define precision, recall, and F1 score:
Statistical analysis
We estimated the 95% confidence intervals (95% CI) and performed hypothesis tests for precision, recall, and F1 using bootstrapping.27 Hypothesis tests are one-sided with an alpha of 0.05. Details are described in the Supplementary Implementation Details section.
Computation
All experiments were implemented with our published Python package, LLM-IE version 1.2.1.25 All LLM inference instances were served with vLLM inference engine version 0.17.028 on a local server instance with four NVIDIA H100 graphical processing units (GPUs).
Results
Summary of dataset and manual annotation
The initial review of 50 clinical notes showed a low agreement between the two cardio-oncology research trainees (Cohen’s kappa of 0.3750; average F1 score of 0.5406 for entity extraction and 0.4676 for entity and attribute end-to-end extraction). The final review conducted by the senior physician resolved the inconsistencies, corrected mislabels, and added missed annotations. A total of 538 myocarditis-related entities and 588 attributes were included in the ground truth dataset (Table 2).
Comparison between the proposed large language model method and cardio-oncology research trainees
For myocarditis entity extraction and entity-attribute end-to-end extraction, our proposed method (i.e. GPT-OSS with sentence-based context engineering and hierarchical prompting) showed F1 scores of 0.7804 [95% CI = (0.7298–0.8311)] and 0.6332 [95% CI = (0.5827–0.6837)], respectively, which are higher than the cardio-oncology trainee annotators’ F1 scores in our study (entity extraction F1 = 0.7386, P = 0.0567; entity-attribute end-to-end extraction F1 = 0.4933, P < 0.001). The estimated time for a trainee annotator to label 1000 clinical notes is 72 h (based on timestamps from the manual annotation performed in our study). While using our proposed method on our local GPU server, it only took about 37 min, which is 116 times faster. Using the minimum wage in the USA in 2024 ($7.25 USD per hour), hiring an annotator to label 1000 clinical notes would cost at least $522 USD. Using the API rate in July of 2026 from cloud computing vendor, OpenRouter,29 we estimated the cost for annotating 1000 clinical notes with our method is $2.04 USD, which is 256 times cheaper (Figure 2).
Results overview. (A) Myocarditis entity extraction. For extracting spans of diagnosis, signs and symptoms, outcomes, imaging, treatments, and procedures, our proposed method, which uses hierarchical prompting with a reasoning large language model, outperforms the average performance of cardio-oncology research trainees. (B) Myocarditis attribute extraction. For extracting entity types, assertions, and dates, our method significantly dominates. (C) Hours to annotate 1000 notes. Our method annotates 1000 clinical notes in under 1 h, which is about 116 times faster than the average cardio-oncology research trainees. (D) Dollars to annotate 1000 notes. Our method costs $2.04 USD (estimated by input and output tokens and the API rate from OpenRouter in August 2025), which is 256 times cheaper than hiring human annotators (cost estimated by the 2024 US minimum wage $7.25 USD).
Performance of the proposed large language model method stratified by entity classes and attributes
Stratified by myocarditis entity classes, our proposed method (i.e. GPT-OSS with sentence-based context engineering and hierarchical prompting) showed higher performance for signs and symptoms [F1 = 0.8201, 95% CI = (0.7591–0.8812)], diagnosis [F1 = 0.8602, 95% CI = (0.7500–0.9704)], and procedure [F1 = 0.8182, 95% CI = (0.6879–0.9485)], while having lower performance for EKG/telemetry findings [F1 = 0.5652, 95% CI = (0.3188–0.8117)] and treatment [F1 = 0.7649, 95% CI = (0.7045–0.8253)]. The outcome, ECHO, and cardiac MRI have few entities in the test set and could not be accurately evaluated. For myocarditis attributes, our proposed method showed higher performance in assertion [F1 = 0.7701, 95% CI = (0.7202–0.8201)] and entity type [F1 = 0.6885, 95% CI = (0.6233–0.7538)] extraction (Table 3).
Table 3Evaluation metrics of the best-performing (reasoning large language model with sentence-based context engineering and hierarchical prompting) model by myocarditis entities and attributesEntity extractionEntity class n Precision (95% CI)Recall (95% CI)F1 (95% CI)Signs and symptoms1350.8028 (0.7223–0.8834)0.8382 (0.7755–0.9010)0.8201 (0.7591–0.8812)Diagnosis410.7692 (0.5986–0.9399)0.9756 (0.9206–1.0306)0.8602 (0.7500–0.9704)Outcome91.0000 (0.3517–1.6483)0.2222 (−0.0761–0.5205)0.3636 (−0.0075–0.7348)EKG/telemetry findings331.0000 (0.9661–1.0339)0.3939 (0.1573–0.6306)0.5652 (0.3188–0.8117)ECHO41.0000 (1.0000–1.0000)1.0000 (1.0000–1.0000)1.0000 (1.0000–1.0000)Cardiac MRI41.0000 (0.0777–1.9223)0.2500 (−0.3394–0.8394)0.4000 (−0.2425–1.0425)Procedure240.9474 (0.8624–1.0324)0.7200 (0.5384–0.9016)0.8182 (0.6879–0.9485)Treatment1950.8544 (0.8070–0.9019)0.6923 (0.6080–0.7766)0.7649 (0.7045–0.8253) Micro average 0.8363 (0.7866–0.8860)0.7315 (0.6702–0.7929)0.7804 (0.7298–0.8311)Entity and attribute extraction (end-to-end)Attribute types n Precision (95% CI)Recall (95% CI)F1 (95% CI)Entity type4450.7259 (0.6618–0.7901)0.6548 (0.5791–0.7305)0.6885 (0.6233–0.7538)Assertion4120.8166 (0.7671–0.8661)0.7287 (0.6669–0.7905)0.7701 (0.7202–0.8201)Date occurred370.3811 (0.3135–0.4488)0.3370 (0.2676–0.4063)0.3577 (0.2907–0.4247)Date reported160.5644 (0.2750–0.8538)0.5182 (0.2476–0.7888)0.5403 (0.2619–0.8186) Micro average 0.6699 (0.6225–0.7172)0.6003 (0.5407–0.6600)0.6332 (0.5827–0.6837)EKG, electrocardiogram; ECHO, echocardiogram; MRI, magnetic resonance imaging.
Evaluation metrics of the best-performing (reasoning large language model with sentence-based context engineering and hierarchical prompting) model by myocarditis entities and attributes
EKG, electrocardiogram; ECHO, echocardiogram; MRI, magnetic resonance imaging.
Systematic evaluation of reasoning large language model, context engineering, and hierarchical prompting
Comparing different experimental settings, our proposed methods showed an overall improvement in both myocarditis entity extraction and entity-attribute end-to-end extraction (Figure 3 and Supplementary material online, Table S6). This section details the comparisons of reasoning LLM, context engineering, and hierarchical prompting methods.
Experiments comparing different large language model prompting methods. (A) Entity class extraction shows that our proposed system, combining context engineering, hierarchical prompting, and a reasoning large language model, outperforms the average of cardio-oncology research trainees. (B) Entity and attribute extraction shows that, using our proposed context engineering and hierarchical prompting methods, both Medgemma and GPT-OSS significantly outperform the average of research trainees.
Reasoning LLM. The reasoning LLM, GPT-OSS, showed dramatic improvements with context engineering, regardless of one-step or hierarchical prompting methods. However, when prompted with the entire document as context (without context engineering), the medical LLM, Medgemma, dominated.
Reasoning LLM. The reasoning LLM, GPT-OSS, showed dramatic improvements with context engineering, regardless of one-step or hierarchical prompting methods. However, when prompted with the entire document as context (without context engineering), the medical LLM, Medgemma, dominated.
Context engineering. Under the same configuration, the paragraph-based context engineering showed an overall benefit. Fixing the prompting method to ‘one-step’ (non-hierarchical), Llama-4-Scout, Medgemma, and GPT-OSS with ‘Paragraph-based’ context engineering showed 0.1147, 0.1538, and 0.3113 increase in the entity extraction F1 and 0.0806, 0.1335, and 0.2668 increase in the end-to-end F1. With the ‘hierarchical’ prompting method, Llama-4-Scout, Medgemma, and GPT-OSS, with ‘Paragraph-based’ context engineering, showed 0.1231, 0.2029, and 0.2761 increase in the entity extraction F1 and 0.1054, 0.1628, and 0.2240 increase in the end-to-end F1. However, the best-performing system was GPT-OSS with sentence-based context engineering and hierarchical prompting (F1 = 0.6332 end-to-end; proposed method).
Context engineering. Under the same configuration, the paragraph-based context engineering showed an overall benefit. Fixing the prompting method to ‘one-step’ (non-hierarchical), Llama-4-Scout, Medgemma, and GPT-OSS with ‘Paragraph-based’ context engineering showed 0.1147, 0.1538, and 0.3113 increase in the entity extraction F1 and 0.0806, 0.1335, and 0.2668 increase in the end-to-end F1. With the ‘hierarchical’ prompting method, Llama-4-Scout, Medgemma, and GPT-OSS, with ‘Paragraph-based’ context engineering, showed 0.1231, 0.2029, and 0.2761 increase in the entity extraction F1 and 0.1054, 0.1628, and 0.2240 increase in the end-to-end F1. However, the best-performing system was GPT-OSS with sentence-based context engineering and hierarchical prompting (F1 = 0.6332 end-to-end; proposed method).
Hierarchical prompting. With the same configuration, our hierarchical prompting method showed a consistent benefit across the board. Fixing the context engineering method to ‘document-as-context’, Llama-4-Scout, Medgemma, and GPT-OSS with hierarchical prompting showed 0.1816, 0.0988, and 0.1114 increases in the entity extraction F1 and 0.1221, 0.0810, and 0.1091 increases in the end-to-end F1. With the ‘Paragraph-based’ context engineering, Llama-4-Scout, Medgemma, and GPT-OSS with hierarchical prompting showed 0.1900, 0.1479, and 0.0762 increases in the entity extraction F1 and 0.1469, 0.1103, and 0.0663 increases in the end-to-end F1.
Hierarchical prompting. With the same configuration, our hierarchical prompting method showed a consistent benefit across the board. Fixing the context engineering method to ‘document-as-context’, Llama-4-Scout, Medgemma, and GPT-OSS with hierarchical prompting showed 0.1816, 0.0988, and 0.1114 increases in the entity extraction F1 and 0.1221, 0.0810, and 0.1091 increases in the end-to-end F1. With the ‘Paragraph-based’ context engineering, Llama-4-Scout, Medgemma, and GPT-OSS with hierarchical prompting showed 0.1900, 0.1479, and 0.0762 increases in the entity extraction F1 and 0.1469, 0.1103, and 0.0663 increases in the end-to-end F1.
Scalability
To highlight the scalability of our proposed method, we deployed a structured data generation pipeline on 20 patients, with 10 developing ICI-associated myocarditis and 10 not, resulting in a total of 2619 clinical notes collected 90 days before and after the initial ICI. Table 4 highlights the automatically extracted entities and attributes for various data points of interest. Of particular importance for determining patients with worse outcomes was the identification of major adverse cardiovascular events such as life-threatening arrhythmias, heart failure, and stroke. Additionally, our method allows for detecting diagnostic nuances such as the presence or absence of inflammatory infiltrate/or myocyte death on endomyocardial biopsy reports. Lastly, this method also allows for identifying not only if patients had imaging or diagnostic procedures, but also if these procedures were planned for but then cancelled or never performed.
Table 4Myocarditis evidence extraction of 2619 clinical notes from 20 patients using our proposed method (reasoning large language model with sentence-based context engineering and hierarchical prompting)Myocarditis entity extractionEntity classEntity typeMentionsMentions (non-myocarditis patients)Mentions (myocarditis patients)Cardiac MRICardiac MRI1771176DiagnosisCerebrovascular Accident (Stroke or TIA)803743Congestive Heart Failure80674Life-threatening arrhythmia7302728Lower limb ischemia909Myocardial infarction12810118Myocarditis177591766Pericarditis34133Pulmonary embolism21030180Venous thrombosis39861337ECHOPericardial effusion1523149WMA40139EKG/telemetry findingsAtrial fibrillation or flutter1673164Complete AV block2790279Left anterior fascicular block78078Left bundle branch block19019Non-sustained ventricular tachycardia2580258Right bundle branch block2460246Sinus rhythm40115386Sustained ventricular tachycardia606OutcomeLVEF1483145cardiac symptoms505conduction disorder1140114pericardial effusion909ProcedureLHC/EMB7970797Signs and symptomsCardiogenic shock52349Chest pain940266674Cough540208332Dyspnoea26015752026Fatigue1185190995Fever16445321112Lower limb oedema737193544Orthopnea15712145Palpitations677101576Paroxysmal nocturnal dyspnoea36135Presyncope20641165Syncope701456Weight gain801466TreatmentAbatacept1400140Anti B-cell therapy4510451Conventional DMARDs18018Electrical cardioversion202Fixed lead temporary pacemaker303IL-6 inhibitor1560156Inotropes101Invasive ventilation22319Non-invasive ventilation7300730Oxygen therapy55752505PCI or CABG514Pericardiocentesis110Permanent pacemaker40040Pharmacologic cardioversion13013Pharmacologic pacing16016Plasmapheresis8653862Presentation management18373221515Steroids32142682946TNF-a inhibitors2056199Transcutaneous pacing30030Transvenous temporary pacer1350135Upgrade level of care1631162Vasopressors63063Myocarditis attribute extractionValueMentionsMentions (non-myocarditis patients)Mentions (myocarditis patients)AssertionPresence16 608165114 957Absence515811773981Possible866103763Date occurred12 357162410 733Date reported740616875719Date intervention started56483505298Date intervention ended1369451324Procedure statusCompleted7290729Cancelled101Planned63063Other404LHC result2920292EMB resultCMC death808Inflammatory infiltrate1040104CMC death and Inflammatory infiltrate2390239MRI, magnetic resonance imaging; ECHO, echocardiogram; WMA, wall motion abnormality; EKG, electrocardiogram; AV, atrioventricular; LVEF, left ventricular ejection fraction; LHC, left heart catheterization; EMB, endomyocardial biopsy; DMARDs, disease-modifying antirheumatic drugs; IL, interleukin; PCI, percutaneous coronary intervention; CABG, coronary artery bypass graft; TNF, tumour necrosis factor; CMC, cardiomyocyte.
Myocarditis evidence extraction of 2619 clinical notes from 20 patients using our proposed method (reasoning large language model with sentence-based context engineering and hierarchical prompting)
MRI, magnetic resonance imaging; ECHO, echocardiogram; WMA, wall motion abnormality; EKG, electrocardiogram; AV, atrioventricular; LVEF, left ventricular ejection fraction; LHC, left heart catheterization; EMB, endomyocardial biopsy; DMARDs, disease-modifying antirheumatic drugs; IL, interleukin; PCI, percutaneous coronary intervention; CABG, coronary artery bypass graft; TNF, tumour necrosis factor; CMC, cardiomyocyte.
Discussion
In the current landscape of electronic health records, certain clinical data elements and their metadata, like laboratory values and vitals, are already captured and stored in a structured format within the backend of the system. This allows for their direct and reliable retrieval using standard database queries without the need for NLP or LLM-based extraction. In contrast, key clinical information such as signs and symptoms, narrative descriptions of diagnostic procedures and findings (e.g. coronary angiography reports, pathology interpretations, and advanced cardiac imaging), and treatment interventions are often documented exclusively within free-text clinical notes. These unstructured data elements are more challenging to extract at scale using traditional methods.
In this pilot study, we developed an NLP pipeline adopting the latest advancements in LLMs for myocarditis evidence extraction from clinical notes. Our proposed method achieved significantly improved accuracy over manual extraction, while requiring dramatically reduced time and resources. On the entity classes of signs and symptoms (F1 = 0.8201), diagnosis (F1 = 0.8602), and procedure (F1 = 0.8182), and the assertion attribute (F1 = 0.7701), our proposed method showed high performance that is adequate to support downstream clinical research. To explore the optimal use of LLMs, we systematically evaluated the latest reasoning LLM and two practical prompting strategies: context engineering and hierarchical prompting. Our results provide an efficient working example for the cardio-oncology field. With appropriate adjustments in the information extraction schema and prompt templates, our method is generalizable for other medical domains.
A common challenge in NLP pilot studies, including this project, is the trade-off between the rapid pace of LLM advancement and the substantial manual effort required to build adequately powered, generalizable datasets with ground truth. The relatively low sample size and patient cohort size reduce (a) statistical power, (b) generalizability, (c) isolation of training and test sets (i.e. data leakage), and (d) diversity in medical content. We provided an in-depth discussion in the Limitations and Future Directions section. To launch a comprehensive study, more resources and workforce are needed for annotation. Particularly, a large patient cohort is needed to ensure richness of clinical content and to improve generalizability. A patient-level isolation is needed to sample for the training set and test set to avoid data leakage. However, given the timeliness of AI advancement and the promising potential we observed in this pilot study, we believe our results are relevant to both the clinical informatics and cardio-oncology communities.
The use of NLP and LLMs for post-ICI irAEs is a promising approach. Gupta et al.30 proposed a pipeline utilizing machine learning-based NLP approaches to identify cancer patients who develop irAEs after initiating ICI treatment and achieved an overall F1 score of 0.75 across toxicities. A recent study leveraged LLMs to identify and characterize ICI-induced myocarditis, encephalitis, pneumonitis, and severe cutaneous adverse reactions. The authors concluded that the NLP approach is a more robust method for determining ICI-related toxicities compared with the diagnosis code approach and enables additional treatment information to characterize the severity of toxicities.31
Distinguished from previous studies that prompt LLMs for irAEs detection at the patient level (i.e. cohort identification), our work highlights the granular extraction of myocarditis-related clinical concepts (e.g. diagnostic testing results, imaging, treatments, and procedures) and attributes (e.g. subtype, date, and assertion) to construct structured datasets, instead of providing the binary absence or presence of a given irAE diagnosis. As safety and transparency in medical AI raise increased concerns, we proposed a responsible use of AI that utilizes LLMs to collect verifiable evidence to empower physicians for informed decision-making. Furthermore, as clinical guidelines update periodically, keeping AI systems up-to-date has been challenging. This approach still requires establishing eligibility and exclusion criteria for patient populations, the time frame of retrospective analysis, and specifying the type of note to be analysed. A model like ours can then move in and construct a data set composed of key clinical elements that either support or argue against the presence of an event, minimizing the navigation of lengthy, heterogeneous clinical notes to only select patient scenarios. Our approach, which extracts granular data points, preserves clinical research oversight for the diagnosis and treatment of myocarditis while leveraging AI to reduce the burden of chart review, providing a sustainable workflow.
Classically, manual data collection has been considered the gold-standard but as evidenced by the low inter-annotator F1 score between trainees, manual data collection has limitations. The reason for this may be related to the review of lengthy, unstructured clinical notes, causing human fatigue or selective attention, and variability in note interpretation. The use of NLP pipelines using LLM models has the potential to allow providers to perform research, quality improvement, and improve clinical care workflows in a fashion that bypasses the time-consuming step of manual data collection and data cleaning often required prior to analysis. When it comes to rare toxicities, such as ICI-associated myocarditis, use of standardized structured data generation is necessary when working with multi-centre institutions to combine cases into large registries. As a pilot study, our AI model would need to be validated on a larger cohort with both myocarditis-positive and -negative cases, and at institutions outside of MD Anderson to make this a scalable approach for generating structured data points for ICI-associated myocarditis from institutions across the world. To adopt our method in practice, a few engineering aspects need to be addressed. First, LLM inference is computationally demanding, especially for GPUs. We conducted our experiments on a server instance with four NVIDIA H100 GPUs, which is not the most affordable setting. However, in a side experiment, we benchmarked the GPU memory requirement and found our method feasible on low-resource configurations with a minimal total GPU memory of 80 GB (see Supplementary Implementation Details section). Cloud LLM inferencing services and proprietary LLMs with Health Insurance Portability and Accountability Act (HIPAA) compliance provide another option. For example, the Microsoft Azure OpenAI platform can be integrated with our method. Secondly, the development and implementation of LLM-based systems requires expertise in NLP. To mitigate this gap, we open-sourced our recipe for this study: the LLM context engineering can be streamlined with our Python package, LLM-IE25,32; the multi-agent hierarchical prompting design is detailed in the Supplementary material online, Figure S1 and Supplementary material online, Table S4. Finally, like every NLP system, our method needs to be prospectively validated before application. Continuous monitoring would be needed due to potential data drift in practice.
Limitations and future directions
Despite the overall benefits, this study, being a pilot, has a few limitations: First, because of the limited manual annotation workforce, our sample size, especially the test set, is relatively small, causing challenges in a precise estimation of model performance (i.e. wide confidence intervals). Fortunately, with a size of 445 entities in the test set, our results showed a statistically significant difference from manual annotation. Secondly, the test set was sampled from a random subset of 10 patients, which does not fully satisfy the statistical independence assumption, as the clinical notes are nested within a few patients. Furthermore, this limits the diversity of clinical content in the test set. It is hard to analyse whether the model recognizes global, generalizable patterns vs. specific patterns for these patients. Future work is needed to evaluate more diverse patient populations. Similarly, this also limits the isolation of training (i.e. few-shot learning) and testing. Despite having a clean separation of clinical notes in the training set and the test set, information from the sample patient might have been seen by the model in the few-shot examples, causing concerns of data leakage. Future studies with a larger patient cohort should separate the training set and test sets at the patient level. Thirdly, the average performance of two cardio-oncology research trainees in our study does not fully represent the average of human annotators. Fourthly, some entity classes (e.g. outcome, ECHO, and cardiac MRI) showed wide confidence intervals, associated with relatively low occurrence in the test set. The entity type skewness is a universal challenge in clinical NLP. In a large-scale clinical NLP dataset with over 500 annotated clinical notes and 59 000 entities, the minority entity type (i.e. medication duration) accounts for only 1.8% of all entities in the test set.26 This also points to the performance differences observed among different entity categories in our study. For these reasons, it is important to recognize this is only a pilot study and large-scale deployment of this model for data extraction without human supervision is still premature. Finally, we evaluated our method on a single-centre data source, which, despite being a large cancer centre with over 25 000 employees, could be limited to a certain writing style, abbreviations, and typical semantic structures. Future work on external validation is needed.
Documentation practices vary widely across providers and institutions. Our model was trained exclusively on documentation patterns from a single centre, reflecting local style and structure characteristics of notes. Future development of LLM-based extraction systems should incorporate training data drawn from multiple institutions to capture the complexity and variability of documentation practice.
Once a model with strong internal and external validity has been established, the challenge of which notes to process and how many remains. The cost and compute time increase proportionally with note volume. A rapid query of our institutional database yielded a total of 6792 progress notes across 100 random patients within 365 days of having started their ICI treatment plan at MD Anderson, illustrating the magnitude of data that these systems may need to handle. An important area of opportunity is the development of efficient note-triage frameworks to narrow the subset of notes requiring extraction.
Our model can also be improved by expanding the range of clinical data elements extracted that are relevant for ICI-myocarditis patients, such as electromyography findings, respiratory mechanics evaluations, and respiratory outcomes. Incorporating these into the existing extraction schema may facilitate the identification of predictors of ICI-myocarditis incidence, severity, and clinical outcomes. Finally, these methods can be broadly applied beyond ICI-associated myocarditis. Within cardio-oncology and in other cardiovascular and immunotherapy domains, we plan to train and develop additional models capable of extracting clinical evidence for a wider range of adverse and cardiovascular events.
Conclusions
We proposed a practical LLM-based method for ICI-associated myocarditis evidence extraction, achieving improved accuracy while reducing processing time and cost. The results demonstrated the applicability and scalability of LLMs in this domain, although this was a pilot study and human supervision of LLM-based data extraction is still needed. Our proposed method extracts granular, structured data with clinical nuances from clinical notes to support physicians in making decisions on diagnosis and treatment.
Supplementary Material
Contributor Information
Enshuo Hsu, Enterprise Development and Integration, University of Texas MD Anderson Cancer Center, Houston, TX, USA; McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USA.
Sheng-Chieh Lu, Department of Cardiology, University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd, Unit 1451, Houston, TX, USA.
Sara Ebrahimi, Department of Cardiology, University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd, Unit 1451, Houston, TX, USA.
Noah Beinart, Department of Cardiology, University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd, Unit 1451, Houston, TX, USA.
Keila C Ostos-Mendoza, Department of Cardiology, University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd, Unit 1451, Houston, TX, USA.
Anita Deswal, Department of Cardiology, University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd, Unit 1451, Houston, TX, USA.
Kirk Roberts, McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USA.
Trey Kell, Enterprise Development and Integration, University of Texas MD Anderson Cancer Center, Houston, TX, USA.
Jeff Jin, Enterprise Development and Integration, University of Texas MD Anderson Cancer Center, Houston, TX, USA.
Nicolas L Palaskas, Department of Cardiology, University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd, Unit 1451, Houston, TX, USA.
Supplementary material
Supplementary material is available at European Heart Journal – Digital Health.
Author contributions
Enshuo Hsu (Conceptualization, Methodology, Project administration, Writing—original draft, Writing—review & editing [equal], Data curation, Formal analysis [lead]), Sheng-Chieh Lu (Conceptualization, Data curation, Methodology, Validation, Writing—original draft, Writing—review & editing [supporting]), Sara Ebrahimi (Data curation [equal], Writing—original draft, Writing—review & editing [supporting]), Noah Beinart (Data curation [equal], Writing—original draft, Writing—review & editing [supporting]), Keila C. Ostos-Mendoza (Data curation, Formal analysis, Methodology, Validation, Writing—original draft, Writing—review & editing [supporting]), Anita Deswal (Conceptualization, Methodology, Project administration, Supervision, Writing—review & editing [supporting]), Kirk Roberts (Conceptualization, Methodology, Project administration, Supervision, Writing—review & editing [supporting]), Trey Kell (Conceptualization, Methodology, Project administration, Writing—review & editing [supporting]), Jeff Jin (Conceptualization, Methodology, Project administration, Supervision, Writing—review & editing [supporting]), and Nicolas L. Palaskas (Conceptualization, Data curation, Formal analysis, Methodology, Writing—original draft, Writing—review & editing [equal], Project administration, Supervision [lead])
Funding
No funding was used specifically for this project.
Data availability
The data underlying this article will be shared on reasonable request to the corresponding author.
References
Associated Data
Supplementary Materials
Data Availability Statement
The data underlying this article will be shared on reasonable request to the corresponding author.