Prototype-Based Knowledge Guidance for Fine-Grained Structured Radiology Reporting

Chantal Pellegrini, Adrian Delchev, Ege Özsoy, Nassir Navab, Matthias Keicher · 2026

Read Prototype-Based Knowledge Guidance for Fine-Grained Structured Radiology Reporting by Chantal Pellegrini, Adrian Delchev, Ege Özsoy, Nassir Navab, Matthias Keicher (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11938.txt.

Introduction

Pellegrini and co-authors present ProtoSR, a method that fills structured radiology reports with help from free-text knowledge. Structured reports are faster and steadier, but filling them means many fine choices about rare findings. Free-text reports, written at scale in daily care, carry that fine detail. An instruction-tuned model mines over 80,000 MIMIC-CXR studies into a knowledge base, one visual prototype per answer option. ProtoSR retrieves matching prototypes and lets them correct its predictions. On Rad-ReStruct it tops prior methods, most on detailed attribute questions.

Catalog record

CREATORChantal Pellegrini, Adrian Delchev, Ege Özsoy, Nassir Navab, Matthias Keicher
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21078
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11938

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prototype-Based Knowledge Guidance for Fine-Grained Structured Radiology Reporting Chantal Pellegrini, Adrian Delchev, Ege Özsoy, Nassir Navab, Matthias Keicher Abstract Structured radiology reporting promises faster, more consistent communication than free text, but automation remains difficult as models must make many fine-grained, discrete decisions about rare findings and attributes from limited structured supervision. In contrast, free-text reports are produced at scale in routine care and implicitly encode fine-grained, image-linked information through detailed descriptions. To leverage this unstructured knowledge, we propose ProtoSR, an approach for injecting free-text information into structured report population. First, we introduce an automatic extraction pipeline that uses an instruction-tuned LLM to mine 80k+ MIMIC-CXR studies and build a multimodal knowledge base aligned with a structured reporting template, representing each answer option with a visual prototype. Using this knowledge base, ProtoSR is trained to retrieve prototypes relevant for the current image-question pair and augment the model predictions through a prototype-conditioned residual, providing a data-driven second opinion that selectively corrects predictions. On the Rad-ReStruct benchmark, ProtoSR achieves state-of-the-art results, with the largest improvements on detailed attribute questions, demonstrating the value of integrating free-text derived signal for fine-grained image understanding. We will publish our data generation and model code upon acceptance. 1 Introduction Medical imaging is central to diagnosis and clinical decision making, and radiologists communicate findings primarily through radiology reports, typically written in free text. However, free-text reporting is time-consuming and often non-standardized, leading to variability in completeness, and clarity [ 15 , 6 , 13 ] . In contrast, structured radiology reporting organizes findings into predefined fields and standardized answer options, rather than relying solely on unconstrained narrative text. By using controlled vocabularies and templates, it can improve consistency and completeness while also enabling secondary uses such as quality monitoring and downstream analysis [ 6 , 12 , 8 , 13 ] . With growing demands in radiology, automated radiology report generation has the potential to support reporting workflows, but most work has focused on free-text generation [ 20 , 2 , 14 , 22 , 11 ] . In parallel, generalist medical vision–language models such as MedGemma and CheXagent unify multiple radiology tasks, including report writing and question answering, within a single framework [ 16 , 4 ] . Automated structured reporting (SR) is comparatively less explored: earlier work approached it via prompt-based template population or disease/attribute-specific classifiers [ 15 , 10 , 9 , 3 ] . Rad-ReStruct [ 13 ] introduced a fine-grained SR benchmark with a hierarchical template spanning findings and detailed attributes, and proposed an iterative VQA-style model to populate predefined fields. Context-VQA [ 1 ] further improved structured-report population on Rad-ReStruct by incorporating report-derived context during training. A remaining challenge in automated structured reporting is that fine-grained templates include many rare attributes [ 13 ] , while structured datasets such as Rad-ReStruct are limited in size, providing sparse supervision. On the other hand, large public datasets such as MIMIC-CXR [ 7 ] provide hundreds of thousands of paired chest radiographs and free-text reports, offering broad coverage across common and rare findings. However, these reports differ significantly in style and word choice, complicating direct mapping into a strict SR taxonomy. Recent instruction-tuned LLMs make it increasingly feasible to distill free-text reports into standardized, template-aligned signals [ 21 , 5 ] . This opens the possibility of using large-scale free-text collections as an auxiliary knowledge source to improve image-based structured reporting. Knowledge integration has been explored in free-text reporting in the form of feature-level fusion of e.g. prior patient information or metadata [ 29 , 2 ] and via retrieval of similar cases using prototype memories or knowledge bases [ 19 , 25 , 27 , 17 ] . RadIR [ 28 ] mines fine-grained supervision from free-text radiology reports for scalable retrieval, but does not address how retrieved evidence can be injected into structured prediction pipelines. Overall, these approaches mainly operate in an unstructured output space, whereas SR requires mechanisms that influence fine-grained discrete decisions. To close this gap, we propose ProtoSR, a prototype-conditioned late-fusion framework for fine-grained structured radiology reporting that leverages information extracted from routine free-text reports. ProtoSR transforms paired images and free-text reports into an explicit prototype memory and learns to retrieve prototypes that influence discrete per-field answer selection, enabling targeted corrections of long-tail decisions. We make two main contributions: we propose an LLM-driven mining, normalization, and filtering pipeline that converts a large-scale free-text report collection into a multimodal prototype knowledge base aligned with a structured reporting template and a prototype-conditioned late-fusion module that converts retrieved examples into an answer-aligned correction signal, selectively revising fine-grained predictions while preserving the base model’s overall behavior. Our experiments demonstrate consistent gains, with the strongest improvements on detailed attribute decisions, indicating that routine free-text reports can be leveraged as knowledge signal to improve fine-grained understanding needed for structured reporting. 2 Method Figure 1: Overview of our architecture. The hierarchical SR base model produces base logits, while the prototype-conditioned knowledge branch retrieves label-aligned examples from a prototype bank and converts them into a scaled static residual logit correction. The final prediction is made from the fused logits. Structured reporting datasets are often small and imbalanced, with studies containing diverse fine-grained attributes (e.g., location, visual appearance, severity) being underrepresented. To address this we propose ProtoSR, a structured reporting method, which can leverage knowledge from large free-text report databases. First, we mine free-text radiology reports to recover label-specific visual prototypes aligned to a fixed, fine-grained structured reporting template. We then leverage this knowledge base to retrieve prototype evidence and inject it into a hierarchical structured-reporting vision-language model. 2.1 Knowledge Base Construction Figure 2: Knowledge base extraction. Dataset A defines the target structured reporting template, while Dataset B contains paired images and free-text reports. We first expand the template label vocabulary with alternative phrasings, then identify template-aligned label occurrences in the free-text reports, and finally apply filtering to build a knowledge base that links Dataset B images to the template-aligned labels. We construct a template-aligned prototype knowledge base by mapping a large corpus of paired images and free-text reports (dataset B ) to the fine-grained label space defined by a structured dataset (dataset A ). Dataset A defines the target template with each template answer option corresponding to a label \ell\in L . Exploiting the report–image pairs in B , any study whose report affirms \ell is treated as a candidate exemplar for \ell . The knowledge base construction process shown in Fig. 2 consists of the following steps: Terminology expansion: To improve robustness to reporting variability in B , we expand the vocabulary associated with each target label \ell . We leverage a zero-shot LLM to propose synonyms, abbreviations, and alternative phrasings, yielding a dictionary that maps description variants to the canonical label. Template-constrained extraction: For each report r\in B , we extract template-aligned labels. We first query the LLM to decide whether a finding corresponding to \ell is contained in a report. If present, a subsequent query extracts the corresponding attribute value(s) specified by the template. We apply the extraction hierarchically, querying deeper attributes only when their parent finding is present and use constrained decoding, where only template-aligned answers are kept, to ensure valid outputs. Post-processing and Knowledge

Read the complete text