Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation
Jinghan Sun, Dong Wei, Zhe Xu, Donghuan Lu, Hong Liu, Hong Wang, Sotirios A. Tsaftaris · 2024
Read Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation by Jinghan Sun, Dong Wei, Zhe Xu, Donghuan Lu, Hong Liu, Hong Wang, Sotirios A. Tsaftaris (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13599.txt.
Introduction
Jinghan Sun and co-authors join two chest scan tasks in one frame in a 2024 paper. One task finds and maps odd spots in chest films. The other writes the report that sums those finds for care. Past tools did each alone, though the two feed each other.
Their frame grows both skills in step, each teaching the other. It learns from rich sets with boxes drawn and reports penned, and from weak sets with reports alone. That weak trove is vast and was once left on the shelf. Ward teams gain reports and maps that rise as one, with less hand-marked data to buy.
Catalog record
| CREATOR | Jinghan Sun, Dong Wei, Zhe Xu, Donghuan Lu, Hong Liu, Hong Wang, Sotirios A. Tsaftaris |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13599
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation Thanks: Manuscript received August 24, 2026. This work was supported by National Natural Science Foundation of China (Grant No. 62371409) and Fujian Provincial Natural Science Foundation of China (Grant No. 2023J01005). (J. Sun and D. Wei contributed equally to this work.) (Corresponding authors: L. Wang and Y. Zheng.) Thanks: J. Sun, H. Liu, and L. Wang are with National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen, China (email: [email protected], [email protected], [email protected]). Thanks: D. Wei, D. Lu, H. Wang, and Y. Zheng are with Jarvis Research Center, Tencent YouTu Lab (email: {donwei, caleblu, hazelhwang, yefengzheng}@tencent.com). Thanks: Y. Zheng is also with Medical Artificial Intelligence Laboratory, Westlake University, Hangzhou, China. (email: [email protected]) Thanks: Z. Xu is with the Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China (email: [email protected]). Thanks: Sotirios A. Tsaftaris and Steven McDonagh are with School of Engineering, The University of Edinburgh, Edinburgh, UK (email: {s.tsaftaris, s.mcdonagh}@ed.ac.uk) Thanks: J. Sun, H. Liu, and Z. Xu contributed to this work as interns at Jarvis Research Center, Tencent YouTu Lab. Jinghan Sun, Dong Wei, Zhe Xu, Donghuan Lu, Hong Liu, Hong Wang, Sotirios A. Tsaftaris Abstract Anatomical abnormality detection and report generation of chest X-ray (CXR) are two essential tasks in clinical practice. The former aims at localizing and characterizing cardiopulmonary radiological findings in CXRs, while the latter summarizes the findings in a detailed report for further diagnosis and treatment. Existing methods often focused on either task separately, ignoring their correlation. This work proposes a co-evolutionary abnormality detection and report generation (CoE-DG) framework. The framework utilizes both fully labeled (with bounding box annotations and clinical reports) and weakly labeled (with reports only) data to achieve mutual promotion between the abnormality detection and report generation tasks. Specifically, we introduce a bi-directional information interaction strategy with generator-guided information propagation (GIP) and detector-guided information propagation (DIP). For semi-supervised abnormality detection, GIP takes the informative feature extracted by the generator as an auxiliary input to the detector and uses the generator’s prediction to refine the detector’s pseudo labels. We further propose an intra-image-modal self-adaptive non-maximum suppression module (SA-NMS). This module dynamically rectifies pseudo detection labels generated by the teacher detection model with high-confidence predictions by the student. Inversely, for report generation, DIP takes the abnormalities’ categories and locations predicted by the detector as input and guidance for the generator to improve the generated reports. Finally, a co-evolutionary training strategy is implemented to iteratively conduct GIP and DIP and consistently improve both tasks’ performance. Experimental results on two public CXR datasets demonstrate CoE-DG’s superior performance to several up-to-date object detection, report generation, and unified models. Our code is available at https://github.com/jinghanSunn/CoE-DG. \FAILED \FAILED I Introduction Chest X-ray (CXR) is the most commonly performed diagnostic radiograph in clinics, which helps spot abnormalities and diseases of the airways, blood vessels, heart, and lungs. This diagnostic workflow includes detecting anatomical abnormalities and writing a report to record the findings. Given the complexity and workload of clinical CXR reading, there is a growing interest in developing automated methods for anatomical abnormality detection [ 1 ] and radiology report generation [ 2 ] in CXR—especially using deep neural networks (DNNs) [ 3 , 4 , 5 , 6 , 7 ] . These methods are expected to expedite clinical workflow and reduce observational oversights. Here, the detection task involves both localization ( e.g. , with bounding boxes) and characterization ( e.g. , cardiomegaly) of the abnormalities. However, training accurate DNN-based detection models usually requires large-scale datasets with high-quality per-abnormality annotations, which is costly in time, effort, and expense. Meanwhile, the visual bias problem, where normal regions dominate a CXR image over abnormal, poses a challenge for accurately spotting tiny abnormal regions by the report generation model. To relieve the annotation burden for abnormality detection, a few works [ 8 , 9 , 10 ] proposed semi-supervised object detection and achieved noteworthy advances in the natural image domain. Most of these methods were built on the teacher-student distillation (TSD) paradigm [ 11 ] . In TSD, a teacher model is firstly trained on the labeled data. Then, a student model is trained on both the labeled data with real annotations and the unlabeled data with pseudo labels generated (predicted) by the teacher. However, compared with objects in natural images, the abnormalities in CXR can be subtle and less well-defined with ambiguous boundaries, thus likely introducing great noise to the pseudo labels and eventually leading to suboptimal performance of semi-supervised learning with TSD. To alleviate the visual bias problem in report generation, some approaches [ 6 , 12 ] proposed aligning predefined disease tags and corresponding visual region features to make the networks focus on abnormal regions related to those tags. However, they required domain expertise and extra labor to define and extract the disease tags, and it was still uncertain whether these models accurately identified abnormal regions while generating reports for input CXR images. Furthermore, most report generation methods prioritized literal consistency but ignored clinical efficacy. Lastly, despite the encouraging performance of the above methods in addressing the detection and generation tasks separately, none of them attempted to bridge these two tasks. We are aware of a series of works [ 13 , 14 , 15 , 16 ] aimed at obtaining a unified framework for various vision-language tasks ( e.g. , visual question answering and image captioning) in the natural image domain. However, these methods required data with paired bounding box and caption annotations for training, thus could not utilize weakly labeled images ( i.e. , with captions but no bounding boxes). This work presents a co-evolutionary abnormality detection and report generation (CoE-DG) framework for bridging both tasks in CXR. Text reports describe significant findings in CXRs and are readily available for most archive radiographs. They are a valuable source of image-level supervision signals unique to medical image data. Unlike most existing unified frameworks that overlooked valuable knowledge in the weakly labeled data, we design an iterative optimization method that fully utilizes them, enhancing the information interaction between vision and language tasks and promoting joint performance improvement. This approach includes a report generator guided information propagation (GIP) module for anatomical abnormality detection and an abnormality detector guided information propagation (DIP) module for report generation. Specifically, through GIP, we first enrich the detector’s input features by incorporating the generator’s informative features. Then, based on TSD [ 11 ] , the generator’s auxiliary categorical prediction is used for noise reduction in the pseudo labels of the detection task. Besides the GIP, we additionally propose self-adaptive non-maximum suppression [ 17 ] (SA-NMS) for intra-image-modal refinement in anatomical abnormality detection. The predictions by both the teacher and student detectors go through NMS together to produce new pseudo detection labels for training. In this way, the pseudo labels generated by the teacher detector are dynamically rectified by high-confidence predictions of the student who is getting better as training continues. Reversely, through DIP, the abnormality tokens and location embeddings extracted from the detector’s output serve as input to the generator. Moreover, abnormality categories predicted by the detector are used as pseudo labels