Evaluating Structured Output Robustness of Small Language Models for Open Attribute-Value Extraction from Clinical Notes
Nikita Neveditsin, Pawan Lingras, Vijay Mago · 2025
Read Evaluating Structured Output Robustness of Small Language Models for Open Attribute-Value Extraction from Clinical Notes by Nikita Neveditsin, Pawan Lingras, Vijay Mago (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01810.txt.
Introduction
Clinical notes hold rich facts, but pulling them out as neat name-value pairs is a job where small language models can stumble on format. Nikita Neveditsin, Pawan Lingras, and Vijay Mago test how well such models keep a set output shape. Their 2025 study tries three common formats for the output.
One format, JSON, came out on top for how often a machine could read the result back without error. Sharp prompts and larger models each raised the share of clean output. Long notes and some note types pulled the share down. The error review finds each format fails in its own repeat ways. The work gives plain advice for teams that run models where patient data must stay private: pick the sturdy format, and prompt with care.
Catalog record
| CREATOR | Nikita Neveditsin, Pawan Lingras, Vijay Mago |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01810
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Structured Output Robustness of Small Language Models for Open Attribute-Value Extraction from Clinical Notes Nikita Neveditsin, Pawan Lingras, Vijay Mago Abstract We present a comparative analysis of the parseability of structured outputs generated by small language models for open attribute-value extraction from clinical notes. We evaluate three widely used serialization formats: JSON, YAML, and XML, and find that JSON consistently yields the highest parseability. Structural robustness improves with targeted prompting and larger models, but declines for longer documents and certain note types. Our error analysis identifies recurring format-specific failure patterns. These findings offer practical guidance for selecting serialization formats and designing prompts when deploying language models in privacy-sensitive clinical settings. 1 Introduction Structured information extracted from clinical narratives enhances clinical decision-making, streamlines reporting, and facilitates research database development Wang et al. 2018 ; Garg and Mago 2021 . Small language models (SLMs) Schick and Schütze 2021 can be deployed on local hardware and therefore meet privacy requirements Neveditsin et al. 2025 , but their utility depends on producing outputs that downstream software can parse automatically. This work examines open attribute-value extraction , a task in which an SLM identifies clinically relevant attribute-value pairs without a predefined schema and serializes them in a standard format Etzioni et al. 2008 ; Zheng et al. 2018 ; Li et al. 2023 ; Brinkmann et al. 2025 . We compare three commonly used formats: JSON, YAML, and XML, and assess robustness via parseability , defined as the proportion of outputs that can be successfully validated by a standard parser without manual correction. We further analyze how document length, note type, model size, and extraction scope (open vs. targeted for medications, symptoms, and demographics) affect parseability, and report on common structural failure modes and key interactions among these factors. Our contributions are as follows: (i) to the best of our knowledge, we provide the first comparative analysis of structured output parseability across three widely used serialization formats (JSON, YAML, XML) in the context of open attribute-value extraction from clinical notes; (ii) we demonstrate how model size, prompt specificity, and clinical document characteristics systematically influence structural robustness; (iii) we identify and categorize recurrent structural failure modes, offering practical insights into common format-specific vulnerabilities in SLM-generated outputs. 2 Related Work Prior work on structured information extraction with transformer-based language models has highlighted both their semantic potential and their syntactic fragility. Research in this area can be broadly categorized by its primary evaluation focus: studies that prioritize the semantic accuracy of the extracted content, and those that more directly engage with the technical challenge of ensuring syntactic validity. In high-stakes domains such as clinical medicine, the evaluation emphasis is typically on semantic accuracy. For example, Balasubramanian et al. 2025 evaluated the extraction of 51 features from breast cancer pathology reports by comparing model outputs against expert-annotated gold standards. Similarly, Kadhim et al. 2025 measured the correctness of extracted findings in inflammatory bowel disease reports using F1 scores. In both cases, models like LLaMA-3.3 were assessed primarily on their ability to extract correct clinical content. Syntactic validity, such as whether outputs conformed to a given format, was assumed rather than explicitly evaluated. Other studies, such as Elnashar et al. 2025 , explored prompt design and efficiency trade-offs across JSON, YAML, and hybrid CSV formats using GPT-4o. While they validated attribute-level correctness, structural robustness was not a primary focus. This focus on semantics often coexists with an implicit acknowledgment of the syntactic fragility of unconstrained model outputs. Work in scientific and technical domains has more directly quantified this issue. Dagdelen et al. 2024 , in the context of materials science extraction, noted parse failures under token limits. Schilling-Wilhelmi et al. 2024 advocates constrained decoding to restrict the model’s vocabulary during generation to enforce structural compliance. While this technique improves parseability, Tam et al. 2024 have shown that tighter constraints may also reduce reasoning flexibility, underscoring a trade-off between structural validity and expressiveness. These findings indicate a gap in evaluating the syntactic reliability of structured outputs. Our study addresses this by focusing specifically on parseability as the primary evaluation criterion, using small instruction-tuned models. 3 Methodology 3.1 Models To assess the impact of output format on small language models, we evaluate seven open-weight instruction-tuned models (Table 1 ). Model Vendor Params (B) Ctx. Window Phi-4 Abdin et al. 2024b Microsoft 14 16K Phi-3.5-mini Abdin et al. 2024a Microsoft 3.8 128K Llama-3.2-3B Grattafiori et al. 2024 Meta 3 128K Llama-3.1-8B Grattafiori et al. 2024 Meta 8 128K Mistral-8B Jiang et al. 2023 Mistral AI 8 128K Qwen3-4B Qwen Team 2024 Alibaba 4 32K Qwen3-14B Qwen Team 2024 Alibaba 14 128K Table 1: SLMs evaluated in this study. We selected 7 models from 4 vendors (Microsoft, Meta, Mistral, Alibaba), some of which contributed more than one model. This allowed us to reduce provider-specific bias while also covering a range of model sizes (3–14B parameters) and context window capacities (ranging from 16K to 128K tokens, as shown in Table 1 ). All models are openly available, support local deployment, and are widely used in the open-source community, ensuring relevance, reproducibility, and suitability for privacy-sensitive clinical use. 3.2 Data We use the EHRCon Goldberger et al. 2000 ; Kwon et al. 2025 dataset, a standardized, open, and ethically compliant subset of MIMIC-III Johnson et al. 2016 that supports reproducible research. It includes 105 randomly selected, de-identified clinical notes with 4,101 annotated entities mapped to 13 structured EHR tables. Derived from a large critical care database, EHRCon captures the complexity of real-world clinical documentation. Its public availability and prior ethical clearance make it suitable for secondary analysis without requiring additional ethical review. EHRCon is well-suited for evaluating structural parseability, and its detailed attribute-level annotations offer opportunities for future research on semantic validity, though we do not pursue that direction in this work. The dataset includes three note types: discharge summaries, nursing notes, and physician notes, each with distinct content and length characteristics (Table 2 ). Discharge summaries, the longest (avg. 1300 words, 2700 tokens), provide a comprehensive account of the hospital stay. Physician notes, of moderate length, focus on assessments and treatment plans. Nursing notes, the shortest, document vitals, patient behavior, and routine care. Type # Documents Avg. Words Avg. Tokens Discharge 38 1306.47 2764.46 Nursing 36 490.33 1153.63 Physician 33 669.91 1914.93 Table 2: Descriptive statistics of clinical note types. Token counts are computed by applying each model’s tokenizer to every document and averaging across models from Table 1 . 3.3 Experimental Setup We assess SLMs in two extraction scenarios. The open format scenario prompts the model to extract any medically relevant information it can infer from a note without relying on a predefined schema. This reflects exploratory or retrospective use cases where schema coverage may be incomplete or unavailable. The targeted scenario narrows the prompt to a specific category: medications, symptoms, or demographics. These categories are commonly prioritized in clinical information extraction for their central role in decision support and downstream clinical tasks Sohn et al. 2013 ; Wang et al. 2018 . This allows us to assess whether more constrained prompts yield more structurally consistent outputs. Figure 1 illustrates the overall workflow. A clinical note is processed under one of the two