Beyond Conventional Transformers: The Medical X-ray Attention (MXA) Block for Improved Multi-Label Diagnosis Using Knowledge Distillation

Amit Rand, Hadi Ibrahim · 2025

Read Beyond Conventional Transformers: The Medical X-ray Attention (MXA) Block for Improved Multi-Label Diagnosis Using Knowledge Distillation by Amit Rand, Hadi Ibrahim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02277.txt.

Introduction

Amit Rand and Hadi Ibrahim build a new eye for X-ray reading in a 2025 paper. A chest scan can hold more than one ill at once, so a tool that names all of them in one pass is what clinics need.

Their block of code, named MXA, tunes the gaze of a sight model to the quirks of X-rays. It takes in fine local marks and the wide scene at once. Set inside a lean sight frame and taught by a big teacher model, the tool read a famed chest set with a strong score. The score beat the team's own base tool by a wide step. The authors say the block is the first gaze built just for chest X-rays.

Catalog record

CREATORAmit Rand, Hadi Ibrahim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-02277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Conventional Transformers: The Medical X-ray Attention (MXA) Block for Improved Multi-Label Diagnosis Using Knowledge Distillation Amit Rand, Hadi Ibrahim Abstract Medical imaging, particularly X-ray analysis, often involves detecting multiple conditions simultaneously within a single scan, making multi-label classification crucial for real-world clinical applications. In this paper, we present the Medical X-ray Attention (MXA) block, a novel attention mechanism tailored specifically to address the unique challenges of X-ray abnormality detection. The MXA block enhances traditional Multi-Head Self Attention (MHSA) by integrating a specialized module that efficiently captures both detailed local information and broader global context. To the best of our knowledge, this is the first work to propose a task-specific attention mechanism for diagnosing chest X-rays, as well as to attempt multi-label classification using an Efficient Vision Transformer (EfficientViT). By embedding the MXA block within the EfficientViT architecture and employing knowledge distillation, our proposed model significantly improves performance on the CheXpert dataset, a widely used benchmark for multi-label chest X-ray abnormality detection, with data augmentation methods applied. Our approach achieves an area under the curve (AUC) of 0.85, an absolute improvement of 0.19 compared to our baseline model’s AUC of 0.66, corresponding to a substantial approximate 233% relative improvement above random guessing (AUC = 0.5). 1 1 1 The code and results for this study are available at: https://github.com/Hadi-M-Ibrahim/Beyond-Conventional-Transformers . 1 Introduction Chest X - rays (CXR) are the most common medical images, yet one scan often reveals several co - existing pathologies that automated readers must detect simultaneously. Vision Transformers (ViTs) [ 1 ] excel at global reasoning, but their quadratic self - attention makes deployment on high - resolution CXR costly in both parameters and floating - point operations (FLOPs). Pure convolutional networks, e.g. DenseNet - 121 [ 2 ] , run fast but miss long - range cues spread across the thorax. The clinical setting therefore demands models that (i) learn multi - label dependencies, (ii) highlight subtle local findings, and (iii) fit the limited compute of hospital workstations. Recent efficient transformers, DeiT [ 3 ] , Swin [ 4 ] , EfficientViT [ 5 ] , trim FLOPs via knowledge distillation (KD) or windowed attention, yet none are specialized for medical images where abnormalities occupy just a few pixels. Moreover, public CXR datasets such as CheXpert [ 6 ] remain modest by ViT standards, amplifying over - fitting and class imbalance. We address these gaps with a domain - aware transformer that pairs global self - attention with inexpensive region attention and learns from a calibrated CNN teacher. Contributions. Our work makes three principal contributions: (i) We extend the EfficientViT to multi-label chest-X-ray diagnosis by replacing the cross-entropy objective with a class-balanced BCE with logit loss, enabling a single forward pass to predict all 14 CheXpert labels. (ii) We introduce the lightweight Medical X-ray Attention ( MXA ) block, which pairs dynamic ROI pooling [ 7 ] with channel–spatial gating [ 8 ] in parallel to multi-head self-attention, capturing subtle lesions without increasing asymptotic complexity. (iii) We devise a multi-label knowledge-distillation scheme in which a frozen DenseNet-121 teacher provides soft targets, guiding the transformer on rare pathologies and stabilizing training on the 224 k-image CheXpert corpus [ 9 ] . On CheXpert our EfficientViT-MultiLabel-M5 model reaches 0.85 mean ROC - AUC, improving the M5 EfficientViT baseline by +0.19 while preserving its 5.7G FLOPs footprint. MXA alone delivers half the gain; distillation adds the remainder, confirming their complementary roles. The resulting architecture offers an attractive accuracy - to - efficiency trade - off for point - of - care deployment and illustrates how task - specific attention bridges the gap between generic ViTs and radiologist needs. 2 Related Works Multi - label image classification and CheXNet. Deep CNNs, from AlexNet to EfficientNet, drove single-label ImageNet accuracy upward, but Vision Transformers (ViTs) surpassed them once large-scale data or distillation became available [ 10 , 11 , 12 , 1 , 3 ] . In multi-label chest X-ray diagnosis several findings (e.g., consolidation, edema, cardiomegaly) often co-occur. The per-label sigmoid activation followed by Binary Cross-Entropy (BCE) loss, widely adopted for such tasks, optimizes each label in isolation. This independence assumption disregards inter-label structure and tends to underweight infrequent diseases, leading to poorer AUC on rare classes [ 13 , 14 ] . CheXNet mitigated class imbalance by fine-tuning DenseNet-121 on 112 k NIH scans and achieved radiologist-level pneumonia detection [ 2 ] ; however, its purely convolutional backbone still treated pathologies independently and lost accuracy on external cohorts. Transformer variants such as TransMed explicitly model label dependencies, yet their quadratic self-attention inflates FLOPs, limiting bedside deployment [ 15 ] . Our approach grafts a lightweight Medical X-ray Attention block onto EfficientViT [ 5 ] and distills knowledge from a calibrated DenseNet teacher, preserving CheXNet’s clinical strengths while delivering ViT-level global reasoning at workstation-level cost. CheXpert benchmark. CheXpert offers 224k radiographs from 65k patients with rule - mined, uncertainty - annotated labels for the same 14 findings, providing a stronger test bed than NIH for real - world deployment [ 6 ] . Its public leaderboard has steered advances ranging from label - uncertainty strategies (e.g., U - on - 1, self - training) to loss designs that directly maximize AUC on imbalanced data [ 16 ] . Transformer backbones, including DeiT and Swin, now dominate the leaderboard but often rely on extensive augmentations and multi - crop inference that inflate latency. However, computationally heavy models impede clinical application where GPUs may not be available. We thus adopt CheXpert as our primary benchmark yet target an efficiency regime suitable for clinical integration: EfficientViT halves FLOPs versus Swin while our MXA block and distillation recover the accuracy gap. By evaluating under identical U - 1 labeling and augmentation settings, we isolate the benefit of task - specific attention rather than dataset heuristics. DenseNet - 121 as teacher. DenseNet - 121’s dense skip connections deliver strong feature reuse with only 8M parameters [ 17 ] ; consequently it is the default backbone in many open - source chest - X - ray libraries such as TorchXRayVision. When pretrained on CheXpert the model attains robust AUC across diverse pathologies and exhibits calibrated probability outputs, making it an ideal teacher for KD. Prior student–teacher pairs have compressed DenseNet into MobileNet or ShuffleNet for on - device screening, but rarely into transformers [ 18 , 19 ] . We keep the teacher frozen and distill its logits into an EfficientViT student, showing that soft multi - label distillation narrows the performance gap while letting the transformer exploit global context unavailable to the convolutional teacher. 3 Methods 3.1 Multi-labeling with Efficient Vision Transformers While Cream’s EfficientViT 1 1 1 https://github.com/microsoft/Cream/tree/main/EfficientViT was engineered for single - label tasks (e.g. ImageNet classification) [ 5 ] , CXR diagnosis demands multi - label predictions, as several abnormalities can co - occur. To handle the co-occurrence of multiple pathologies in a single chest X-ray, we adapt EfficientViT’s single-label framework by redesigning its output head and training loss for multi-label prediction. Specifically, we replace the softmax head with a per-class linear layer trained under a class-balanced binary cross-entropy objective, enabling one efficient transformer pass to simultaneously predict all target abnormalities. Label

Read the complete text