S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Framework

Yingshu Li, Yunyi Liu, Zhanyu Wang, Xinyu Liang, Lingqiao Liu, Lei Wang, Luping Zhou · 2025

Read S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Framework by Yingshu Li, Yunyi Liu, Zhanyu Wang, Xinyu Liang, Lingqiao Liu, Lei Wang, Luping Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02082.txt.

Introduction

Seven researchers build a system for structured radiology reports in a paper from 2025. Free-text reports on chest X-rays are often wordy and uneven, which hides key details. Structured reports sort findings into standard fields such as disease, severity, and location. Older tools could only fill fixed labels or templates, leaving gaps. The team built a new data set from a public archive, with rich structured labels.

They trained a language model to write reports in that format. A new scoring method, S-Score, grades both the diagnosis and its details. The score matched human judgments better than older metrics. The authors argue structure plus fair grading can lift report quality in clinics.

Catalog record

CREATORYingshu Li, Yunyi Liu, Zhanyu Wang, Xinyu Liang, Lingqiao Liu, Lei Wang, Luping Zhou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02082

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Framework Yingshu Li, Yunyi Liu, Zhanyu Wang, Xinyu Liang, Lingqiao Liu, Lei Wang, Luping Zhou Abstract Radiology report generation (RRG) for diagnostic images, such as chest X-rays, plays a pivotal role in both clinical practice and AI. Traditional free-text reports suffer from redundancy and inconsistent language, complicating the extraction of critical clinical details. Structured radiology report generation (S-RRG) offers a promising solution by organizing information into standardized, concise formats. However, existing approaches often rely on classification or visual question answering (VQA) pipelines that require predefined label sets and produce only fragmented outputs. Template-based approaches, which generate reports by replacing keywords within fixed sentence patterns, further compromise expressiveness and often omit clinically important details. In this work, we present a novel approach to S-RRG that includes dataset construction, model training, and the introduction of a new evaluation framework. We first create a robust chest X-ray dataset (MIMIC-STRUC) that includes disease names, severity levels, probabilities, and anatomical locations, ensuring that the dataset is both clinically relevant and well-structured. We train an LLM-based model to generate standardized, high-quality reports. To assess the generated reports, we propose a specialized evaluation metric (S-Score) that not only measures disease prediction accuracy but also evaluates the precision of disease-specific details, thus offering a clinically meaningful metric for report quality that focuses on elements critical to clinical decision-making and demonstrates a stronger alignment with human assessments. Our approach highlights the effectiveness of structured reports and the importance of a tailored evaluation metric for S-RRG, providing a more clinically relevant measure of report quality. [style=chinese] [style=chinese] [style=chinese, orcid=0000-0000-0000-0000] [style=chinese] [style=chinese, orcid=0000-0000-0000-0000] [style=chinese, orcid=0000-0000-0000-0000] [style=chinese, orcid=0000-0000-0000-0000] keywords Structure Radiology Report ,Evaluation Metric ,Large Language Models † † credit: Data curation, Writing - Original draft preparation † † corresponding: Corresponding author 1 Introduction Radiology report generation (RRG)—the task of producing diagnostic descriptions from radiographs such as chest X-rays—has become a pivotal topic at the intersection of clinical practice and artificial intelligence [ 16 , 29 , 15 ] . While most existing approaches focus on free-text generation [ 8 , 12 ] , such outputs often contain redundant or stylistically inconsistent language, hindering clinicians’ ability to efficiently extract critical information [ 9 ] . Structured reporting has been proposed as a solution, offering standardized, template-based formats that enhance report clarity, consistency, and completeness [ 24 , 5 ] . Recent efforts in structured radiology report generation (S-RRG) have explored classification-based and template-driven paradigms [ 23 ] , but these often constrain report expressiveness and overlook fine-grained clinical attributes such as severity, probability, and anatomical location [ 13 ] . More recently, large language models (LLMs) [ 27 , 3 ] have been introduced using zero-shot or prompt-based methods [ 20 , 14 , 6 ] ; however, they suffer from limited controllability and reproducibility, with evaluation protocols still in their infancy. In this work, we aim to overcome these limitations by designing a structured dataset and evaluation metrics that more faithfully capture clinically relevant information. Figure 1: Overview of our proposed framework. We begin by (1) transforming free-text radiology reports into structured formats that encode key clinical attributes such as disease name, probability, severity, and anatomical location. These structured reports are represented in a machine-readable JSON format and serve as supervision signals to (2) train a generation-based baseline model. (3) Finally, we evaluate model performance using both structured metrics and natural language generation (NLG) benchmarks. First, a key limitation in current research on structured radiology report generation is the absence of a comprehensive, structured dataset that captures clinically meaningful details—such as disease name, severity, probability, and anatomical location—in a unified format. While efforts like RadReStruct [ 23 ] frame structured reporting as a hierarchical visual question answering (VQA) task, and others like Replace and Report [ 13 ] adopt template-based classification, these approaches either fragment the report into discrete predictions or rely on insufficiently structured outputs, limiting their clinical fidelity. To address this gap, we introduce MIMIC-STRUC, a new dataset developed in collaboration with radiologists that encodes rich, clinically relevant attributes in a predefined, machine-readable JSON format. Unlike prior datasets, MIMIC-STRUC presents full structured reports rather than isolated classification results or piecemeal answers. To evaluate the utility of this format, we generate sentence-based structured reports derived directly from the JSON outputs, enabling assessment with standard NLG metrics. Empirically, we find that models trained to generate structured JSON first, followed by conversion into natural language, outperform those trained to directly produce sentence-based outputs, highlighting the benefits of structured representation for both learning efficiency and output fidelity. Second, there is a lack of specialized evaluation metrics for structured radiology reports. Clinical metrics like Radgraph F1 [ 11 ] , Chexbert F1 [ 25 ] , and Radcliq [ 30 ] require additional training and may be prone to errors when extracting medically relevant content, potentially overlooking important clinical details. To address this, we propose the Structured Report Score (S-Score) , which evaluates disease prediction accuracy and the precision of key details. We validate our metric by calculating its correlation with GPT scores, as studies [ 18 , 17 ] demonstrate that GPT-based evaluations align well with radiologist assessments, showing strong alignment and confirming its clinical relevance. Third, there is currently no established baseline model for generating fully structured radiology reports. Existing structured report generation methods often rely on classification or hierarchical VQA frameworks to predict specific attributes at different levels [ 23 ] , requiring predefined label sets and hand-crafted templates. In contrast, we propose a generation-based approach that directly produces the entire structured report in a single step, without depending on predefined labels. Inspired by recent advances demonstrating the ability of large language models (LLMs) to generate well-formed structured outputs such as JSON [ 26 ] , we train an LLM-based model for structured radiology report generation (S-RRG). Our contributions are summarized as follows and illustrated in Fig. 1 : (1) We construct a new chest X-ray structured report generation dataset, which includes disease names, severity levels, probabilities, and locations, and demonstrate the efficacy of our proposed structured format. (2) We propose the Structured Report Score (S-Score), a new evaluation metric that assesses both disease prediction accuracy and the precision of clinical details. This metric outperforms traditional metrics (e.g., BLEU, ROUGE) and aligns well with radiologist assessments, as demonstrated by its strong Kendall Tau and Spearman correlation with human-like GPT evaluations. (3) We establish a baseline model for structured report generation, comparing traditional transformer-based methods with LLM-based models. Our results demonstrate that LLMs significantly improve accuracy and feasibility, and we propose a new benchmarking task for evaluation. 2 Related Works 2.1 Radiology Report Generation Radiology report generation (RRG) aims to automatically translate medical images, such as chest X-rays into coherent diagnostic reports. An effective radiology report must not only mention the correct diseases but also accurately describe their clinical attributes such as severity, location, and uncertainty [ 10 ] . To this end, early studies employed encoder-decoder architectures, such as memory-driven transformers designed to retain critical information throughout the generation process [ 7 ] . METransformer [ 28 ] advanced this line of work by introducing multiple learnable “expert” tokens into both the encoder

Read the complete text