PHEONA
: An Evaluation Framework for · 2025
Read PHEONA by : An Evaluation Framework for (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19265.txt.
Introduction
Sarah Pungitore, Shashank Yadav, and Vignesh Subbian propose a framework for testing language models on computational phenotyping in a 2025 paper. Phenotyping, the task of finding patients with a given condition from health records, usually demands heavy manual review. Language models could speed it, but few studies had tested them for this work.
Their framework, called PHEONA, sets out context-specific factors to weigh when judging such models. They demonstrate it on concept classification within phenotyping for acute respiratory failure support therapies. On the sample concepts tested, classification accuracy was high. The authors offer the framework as a shared yardstick for future language-model phenotyping research.
Catalog record
| CREATOR | : An Evaluation Framework for |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19265
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PHEONA : An Evaluation Framework for Abstract Abstract Computational phenotyping is essential for biomedical research but often requires significant time and resources, especially since traditional methods typically involve extensive manual data review. While machine learning and natural language processing advancements have helped, further improvements are needed. Few studies have explored using Large Language Models (LLMs) for these tasks despite known advantages of LLMs for text-based tasks. To facilitate further research in this area, we developed an evaluation framework, Evaluation of PHEnotyping for Observational Health Data (PHEONA) , that outlines context-specific considerations. We applied and demonstrated PHEONA on concept classification, a specific task within a broader phenotyping process for Acute Respiratory Failure (ARF) respiratory support therapies. From the sample concepts tested, we achieved high classification accuracy, suggesting the potential for LLM -based methods to improve computational phenotyping processes. Introduction Clinical phenotyping, the process of extracting information from health data to classify patients based on relevant characteristics, is fundamental for biomedical research [ 1 ] . One such type of these processes is electronic or computational phenotyping, which involves defining cohorts algorithmically using data from electronic health records (EHRs) and other relevant information systems [ 2 ] . The resulting phenotypes have been used for many clinical outcomes and conditions supporting a variety of downstream tasks, including recruitment for clinical trials, development of clinical decision support systems, and hospital quality reporting [ 3 , 4 , 5 ] . The process of developing computable phenotypes includes constructing relevant data elements for classification and then applying an algorithm to produce the cohort(s) of interest [ 6 ] . For example, a computable phenotype for Post-Acute Sequelae of SARS-CoV-2 infection (PASC) was developed by first identifying standard clinical concepts for each PASC -related symptom and then applying an algorithm to identify individuals with PASC based on symptom presence relative to SARS-CoV-2 infection using EHR data [ 7 ] . Traditionally, development of computable phenotypes requires time-intensive, manual review of data and definitions, including identification of relevant data and mapping to controlled vocabularies [ 5 ] . From the PASC example, 6{,}569 concepts across 151 symptoms were manually reviewed over several weeks by a clinician expert to determine relevance of each concept to PASC -specific symptoms [ 7 ] . While developments in Natural Language Processing (NLP) and machine learning methods could improve processing and categorization of clinical data, these methods also add additional steps such as model training and hyperparameter tuning which can be similarly time and resource-intensive [ 3 , 5 ] . Thus, there is a need for substantial improvements to traditional computational phenotyping processes, particularly for tasks that require manual review or intervention. As a relatively new addition to biomedical research, Large Language Models (LLMs) introduce a brand new set of text analysis and generation capabilities that could supplement or replace manual review of clinical data elements [ 8 ] . Furthermore, LLMs can be trained on new tasks through prompt engineering alone, which is a more accessible and portable method of model adaptation compared to traditional model retraining or domain adaptation methods [ 9 ] . Therefore, we hypothesize LLMs can transform computational phenotyping, in particular through acceleration and enrichment of manual review through incorporation of LLM -supported processes. To our knowledge, there have been no studies that have explored the application of LLMs to computational phenotyping tasks. Therefore, this study makes the following contributions: 1. We developed PHEONA ( Evaluation of PHEnotyping for Observational Health Data ), a framework that outlines an evaluation mechanism for LLM -based computational phenotyping. 2. We provided an example usage of all components of PHEONA to promote further research and advancements in computational phenotyping processes. 3. We demonstrated a successful application of LLM -based methods to concept classification, a step within a broader phenotyping process that was previously performed solely via manual review. Methods We first discuss how individual components of PHEONA (Figures 1 and 2 ) were constructed, including descriptions of the associated criteria. We then outline how we demonstrated PHEONA on a specific phenotyping use case. Figure 1 : Overview of the components of Evaluation of PHEnotyping for Observational Health Data (PHEONA) . Development of PHEONA Since LLMs have not previously been applied to computational phenotyping tasks, the first component of PHEONA , the resource requirements, was designed to determine when LLM -based methods may provide advantages over traditional phenotyping ones. The resource requirements outline considerations to estimate time, cost, and computational effort and should be considered before development begins. We based the resource requirements on components of Constructive Cost Model (COCOMO) 2.0 , a cost estimation model for software development projects [ 10 ] . Similar to COCOMO , we focused on attributes of the development effort (amount of work required) and development schedule (timeline of work required) when considering resource requirements for LLM -based projects. However, unlike COCOMO , we assumed an experienced research team so that development effort is primarily a function of existing hardware and software constraints. Table 1 outlines individual attributes of the development effort and schedule for consideration. Following an assessment of whether LLMs could be considered for a specific task, the second component of PHEONA , model ability, provides criteria to evaluate LLM -specific processes (i.e., prompt engineering) and known drawbacks of LLMs (i.e., hallucination and response instability) [ 11 , 12 , 13 ] . Therefore, model ability includes the following subcategories: 1) Prompt Factors; 2) Model Consistency; and 3) Response Correctness. Prompt factors evaluate the effort required for prompt engineering as well as the corresponding response times for each prompt. Model consistency factors evaluate the ability of the model to provide consistent answers, even if changes are made to the prompt, since LLMs can be sensitive to inputs [ 11 , 12 ] . Additionally, automated response extraction from LLM responses is necessary for large volume requests and responses must be consistent to produce reliable results. Finally, response correctness factors were designed to evaluate overall performance of the model, including the presence of hallucinations. These criteria were most closely related to traditional phenotyping evaluation, where phenotyping performance is measured against ground truths. [ 6 ] Table 2 details the model ability requirements. We note that when evaluating the model ability criteria, a random subset of the entire dataset may be necessary for evaluation, especially for large datasets without previously defined ground truths. Phenotyping Use Case We applied PHEONA to concept classification, a single development step for previously defined computable phenotypes for Acute Respiratory Failure (ARF) respiratory support therapies. Although the phenotypes were discussed in-depth previously, we provide a brief overview to contextualize our work [ 14 ] . The phenotyping algorithm classified individuals based on the respiratory therapies received, specifically Invasive Mechanical Ventilation (IMV) , Noninvasive Positive Pressure Ventilation (NIPPV) , and High-Flow Nasal Insufflation (HFNI) during an individual Intensive Care Unit (ICU) stay. The respiratory therapies were identified by the presence of specific concepts, such as “ Hi Flow NC ” and “ BiPAP/CPAP ”, that indicated therapy use. Additionally, for IMV , presence of sedation and paralysis medications was required to verify patients underwent intubation. In the original study, manual review of the data elements was performed to determine which concepts