ClinStructor: AI-Powered Structuring of Unstructured Clinical Texts

Karthikeyan K, Raghuveer Thirukovalluru, David Carlson · 2025

Read ClinStructor: AI-Powered Structuring of Unstructured Clinical Texts by Karthikeyan K, Raghuveer Thirukovalluru, David Carlson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11883.txt.

Introduction

Karthikeyan K, Raghuveer Thirukovalluru, and David Carlson present ClinStructor, a pipeline that converts messy clinical free-text notes into structured question-answer pairs before any prediction model sees them. Raw notes carry hidden biases and format quirks that make models brittle across hospital systems and hard to interpret.

Structuring first buys transparency and control at a modest price: on intensive care mortality prediction, accuracy dropped by only two to three percent in area under the curve compared with direct fine-tuning. Clinicians and auditors can inspect the structured record behind a prediction. For medical AI that must travel between electronic record systems, that trade looks worthwhile.

Catalog record

CREATORKarthikeyan K, Raghuveer Thirukovalluru, David Carlson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11883

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ClinStructor: AI-Powered Structuring of Unstructured Clinical Texts Karthikeyan K, Raghuveer Thirukovalluru, David Carlson Abstract Clinical notes contain valuable, context-rich information, but their unstructured format introduces several challenges, including unintended biases (e.g., gender or racial bias), and poor generalization across clinical settings (e.g., models trained on one EHR system may perform poorly on another due to format differences) and poor interpretability. To address these issues, we present ClinStructor, a pipeline that leverages large language models (LLMs) to convert clinical free-text into structured, task-specific question–answer pairs prior to predictive modeling. Our method substantially enhances transparency and controllability and only leads to a modest reduction in predictive performance (a 2–3% drop in AUC), compared to direct fine-tuning, on the ICU mortality prediction task. ClinStructor lays a strong foundation for building reliable, interpretable, and generalizable machine learning models in clinical environments. 1 Introduction Recent advances in Artificial Intelligence (AI) have driven significant progress across various domains, including healthcare ( 5 ; 59 ) , finance ( 60 ; 22 ; 62 ) , law ( 40 ) . However, in high-stakes contexts such as clinical decision-making, requirements extend beyond prediction performance alone. It becomes critical not only to develop reliable predictive models but also to maintain fine-grained control over the attributes influencing predictions. In such scenarios, interpretability is not merely desirable—it is often essential to comply with regulations (e.g., not using protected features such as gender, occupation, etc.) and to foster trust among clinicians, patients, and stakeholders. Clinical notes, a ubiquitous component of electronic health record (EHR) systems, constitute one of the richest sources of clinical information. These free-text notes can encapsulate nuanced details, thus providing invaluable data for predictive modeling. Nonetheless, clinical free-text is inherently unstructured, presenting both significant opportunities and substantial challenges. Notably, clinical notes lack a standardized format, varying extensively in style, structure, and terminology—not only internationally and across languages but even within individual hospitals between different departments or clinicians ( 8 ) . This variability complicates the development of reliable and controllable predictive models. Training predictive models directly on clinical text in a black-box manner raises important concerns. Firstly, there is often insufficient insight into the exact information leveraged by the model to predict an outcome. Clinical notes frequently contain protected attributes, such as race or gender, which – while potentially relevant in certain contexts – may inadvertently introduce biases if not explicitly managed. Additionally, clinical text is susceptible to label leakage, especially for tasks such as mortality prediction, where phrases like “the patient has passed away” or “in critical condition” can directly reveal outcomes. Consequently, models might learn superficial shortcuts, inflating performance metrics without capturing meaningful clinical signals. Moreover, models optimized on a test set from the same data distribution as training may fail to generalize effectively to external datasets from differing institutions, regions, or populations. Variations in documentation practices, terminology, and structure can potentially degrade performance in real-world, cross-site deployments. Figure 1: Illustration of the ClinStructor Pipeline: The proposed pipeline comprises three main stages: (1) Feature Identification, (2) Feature Extraction, and (3) fine-tuning. Several approaches have been proposed to mitigate the challenges posed by unstructured clinical text. These include: (1) bias mitigation methods, which either remove sensitive attributes such as race or gender from input data or transform input to an intermediate representations that obscure sensitive information ( 45 ) ; (2) distributional robustness techniques, explicitly designed to improve resilience against shifts in data distributions across EHR databases or clinical institutes or patient populations ( 47 ) ; and (3) post-hoc explanation techniques, such as highlighting textual spans most influential to predictions ( 52 ) . While valuable, these methods frequently fail to address the root cause of many challenges – the fundamentally unstructured nature of clinical text, making it inherently difficult to ascertain which information the model has utilized. To address the underlying issue of clinical text’s inherent lack of structure, we introduce ClinStructor , illustrated in Figure 1 . ClinStructor is a novel pipeline designed to transform clinical notes into structured representations (question-answer pairs), prior to predictive modeling. Instead of directly fine-tuning models on raw notes, we leverage LLMs to systematically extract meaningful, task-relevant features, in our case, ICU mortality prediction. Specifically, we prompt LLMs to identify a set of clinically relevant questions informed by both clinical domain knowledge and by task-specific data. Subsequently, the same LLMs extract answers to these questions from the patient’s admission notes, resulting in structured question-answer pairs. This effectively converts the unstructured clinical text into structured data, significantly enhancing transparency and controllability. By standardizing the input format, our approach enables thorough inspection of the extracted features, facilitating verification of their clinical relevance and consistency. While subsequent fine-tuning of predictive models on these structured representations slightly diminishes interpretability, it allows leveraging pretrained LLM knowledge for the downstream predictive task. Thus, our approach bridges the expressive richness of unstructured text with the interpretability, reliability, and generalizability inherent in structured data representations. We empirically evaluate ClinStructor against direct fine-tuning on raw clinical notes. Our results indicate a modest performance trade-off, a decrease of approximately 2 to 3% in AUC. The observed performance reduction is anticipated due to potential information loss inherent to structured transformations. Nevertheless, this trade-off delivers substantial advantages. Explicitly defining features used for prediction enables inspection and verification of input signals, effectively mitigating label leakage concerns. Importantly, our approach provides foundational building blocks for constructing predictive models exhibiting enhanced interpretability, robustness, and generalizability—especially crucial when deployed across diverse clinical environments characterized by substantial variability in documentation practices. The remainder of this paper is structured as follows. Section 2 discusses background and related literature. Section 3 details our ClinStructor pipeline, describing each stage comprehensively. Section 4 outlines our experimental setup and presents comparative quantitative results. Further analytical insights are provided in Section 5 . In Section 6 , we highlight our method’s limitations and propose mitigation strategies. Finally, Section 7 summarizes key insights and concludes. The primary contributions of our work include: 1. Introducing ClinStructor , a novel LLM-based pipeline converting unstructured clinical notes into structured, task-specific question-answer pairs. 2. Empirically demonstrating that ClinStructor retains the majority of clinically relevant information, achieving competitive predictive performance in the ICU Mortality prediction task. 3. Discussing crucial limitations of our structured approach, along with detailed suggestions for addressing these issues in future research. 2 Background and Related Works 2.1 Overview of Interpretability Techniques Interpretability is a crucial and growing focus within Machine Learning research ( 30 ; 10 ) . Traditional models, such as linear regression, Support Vector Machines, decision trees, and AdaBoost, inherently provide interpretability for tabular data (numerical and categorical). However, interpretability significantly diminishes with more complex modalities, such as textual or image data, which typically require more sophisticated modeling approaches. For text-based data, several post-hoc methods have been proposed to explain model predictions. Notable examples include SHapley Additive exPlanations (SHAP) ( 27 ) , LIME for Text ( 37 ) , Saliency Maps ( 41 ) , attention visualization techniques ( 52 ; 48 ) , and Integrated Gradients ( 46 ) . These approaches primarily rely on highlighting influential segments of text to explain predictions. However, these explanations may not be faithful or reliable. Alternatively, interpretability can also be achieved through similarity

Read the complete text