Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography
Bowen Shi, Weiwei Cao, Ruifeng Yuan, Wanxing Chang, Wenrui Dai, Hongkai Xiong, Ling Zhang, Jianpeng Zhang · 2026
Read Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography by Bowen Shi, Weiwei Cao, Ruifeng Yuan, Wanxing Chang, Wenrui Dai, Hongkai Xiong, Ling Zhang, Jianpeng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25546.txt.
Introduction
Teaching an AI to read chest CT scans from radiology reports usually aligns whole volumes with whole reports, a blur too coarse for diseases sharing one organ. Bowen Shi and colleagues build a disease centred alternative in 2026. A hybrid encoder pairs a three dimensional convolutional front end, good at local anatomy, with transformer attention for global context. Learnable query tokens then pull disease specific meaning out of each full report and align it with the matching image features.
Diagnosis aware prompts built from real clinical phrases and pooled disease prototypes close the gap between pretraining and use. On two public benchmarks the model lifted diagnostic accuracy by about five points over prior methods. On a harder sixty disease test the gain approached ten points. The pretrained system also transferred well to writing radiology reports, suggesting broad clinical reuse.
Catalog record
| CREATOR | Bowen Shi, Weiwei Cao, Ruifeng Yuan, Wanxing Chang, Wenrui Dai, Hongkai Xiong, Ling Zhang, Jianpeng Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25546
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography Bowen Shi, Weiwei Cao, Ruifeng Yuan, Wanxing Chang, Wenrui Dai, Hongkai Xiong, Ling Zhang, Jianpeng Zhang Abstract Vision–language pre-training (VLP) holds great promise for general-purpose medical AI by leveraging radiology reports as rich textual supervision, yet existing methods struggle with 3D CT imaging due to inefficient visual backbones and coarse semantic alignment. To address these issues, we propose a tailored VLP framework featuring three key components: (1) a CNN–ViT hybrid encoder that replaces ViT’s patch embedding with a 3D CNN backbone to efficiently capture local anatomical details while preserving global attention and compatibility with pre-trained cross-modal priors; (2) a disease-level contrastive learning mechanism using learnable query tokens to dynamically extract disease-specific semantics from full reports and align them with corresponding visual features, thereby disentangling distinct diseases within the same anatomical region; and (3) a diagnosis-aware prompt strategy that employs real clinical phrases and aggregated disease prototypes to bridge the pre-training–inference gap and enhance zero-shot diagnostic reliability. Our model achieves state-of-the-art performance on CT-RATE (84.4% AUC, +5.1%) and Rad-ChestCT (75.4% AUC, +5.4%), with even larger gains (+9.8% AUC) on a challenging 60-disease benchmark, and demonstrates strong transferability to radiology report generation, underscoring the generality and clinical utility of our approach. 1 Introduction Vision–language pre-training (VLP) ( 19 ; 26 ; 20 ; 40 ; 30 ) has introduced a novel paradigm for building general-purpose, versatile medical artificial intelligence systems in recent years. Unlike traditional supervised learning that relies on human-defined, limited categorical labels, VLP leverages clinically generated radiology reports as rich, nuanced linguistic supervision signals to jointly model visual and textual information within a unified semantic space. This paradigm captures semantic associations between images and text, endowing models with zero-shot reasoning and cross-task generalization capabilities, thereby supporting diverse clinical applications such as multi-disease diagnosis and report generation. Despite the promise of medical VLP for general-purpose diagnosis, current methods face two key limitations. First, visual architectures are poorly suited to 3D medical imaging. Most models repurpose backbones originally designed for natural 2D images. ViT-based approaches ( 12 ; 31 ) , when naively extended to 3D computed tomography (CT), suffer from excessive computational cost due to volumetric patchification and often sacrifice spatial resolution, thereby losing critical details of small lesions. In contrast, 3D convolutional neural networks (CNNs) ( 1 ; 3 ) better capture local anatomical textures but deviate from mainstream vision–language model (VLM) architectures ( 26 ; 40 ) , limiting their ability to leverage cross-modal priors learned during large-scale pre-training and hindering effective transfer in joint representation learning. Second, semantic alignment remains coarse. Mainstream methods ( 12 ; 1 ) rely on global image–text contrastive learning, ignoring fine-grained correspondences between localized pathological findings and their textual descriptions. While recent work such as fVLM ( 31 ) aligns representations at the anatomical level, distinct pathologies within the same organ, such as “atelectasis”, “emphysema”, and “lung nodules”, are still grouped under a generic label like “abnormal”. This semantic conflation obscures disease-specific image–language associations, impairing both diagnostic discrimination and model interpretability. To overcome these limitations, medical VLP stands to benefit significantly from advances in general-purpose foundation models. On one hand, modern VLMs offer rich, cross-modal priors—pre-trained visual-textual semantic alignments—that enable visual representations to better interpret clinical language in radiology reports, provided architectural compatibility is maintained. On the other hand, large language models (LLMs) ( 25 ; 11 ; 7 ; 39 ) can automatically extract disease-level labels from unstructured radiology reports, particularly valuable when manual annotations are scarce, thereby enabling fine-grained, disease-centric alignment. We argue that a co-design strategy integrating expressive, VLM-compatible visual representations with LLM-augmented, disease-specific learning objectives is essential to translate the clinical potential of foundation models into effective medical VLP. To this end, we propose a novel VLP framework tailored for 3D CT imaging, centered on the co-design of efficient representation, fine-grained alignment, and robust inference. First, we introduce a CNN–ViT hybrid encoder that replaces the standard ViT patch embedding module with a 3D CNN backbone and integrates multi-scale features. By initializing from a pre-trained ViT ( 9 ) , our hybrid encoder preserves global attention mechanisms while enabling effective integration of cross-modal priors; simultaneously, its 3D CNN backbone efficiently captures local anatomical details across multiple scales, yielding expressive representations for 3D medical volumes. Second, we propose a disease-level contrastive learning mechanism that employs learnable query tokens to dynamically extract disease-specific semantics from full radiology reports and explicitly aligns them with corresponding visual features. This fine-grained alignment disentangles co-occurring pathologies within the same anatomical region, representing the finest-grained semantic alignment reported to date. Crucially, the mechanism flexibly leverages both ground-truth and LLM-extracted disease labels to drive contrastive learning, enabling scalable training while preserving diagnostic granularity. Finally, to bridge the gap between pre-training and zero-shot inference and fully realize the potential of our disease-centric design, we introduce a diagnosis-aware prompt strategy. Rather than relying on handcrafted templates, we construct prompts from real clinical phrases and reuse the learned disease query tokens to extract consistent, condition-specific semantics. By aggregating these semantics across a diverse report collection into global disease prototypes, our approach significantly enhances inference robustness and boosts zero-shot diagnostic reliability. We systematically evaluate our method across multiple benchmarks. On the CT-RATE ( 12 ) test set, our model achieves an AUC of 84.4%, outperforming the current state of the art by 5.1%. This gain remains consistent on the external Rad-ChestCT ( 10 ) test set (+5.4% AUC), underscoring strong generalization. To further assess the scalability of disease-level alignment, we employ Qwen3-Max ( 39 ) to enrich CT-RATE with fine-grained disease annotations, resulting in a challenging diagnostic task spanning 60 distinct conditions. Under this more demanding setting, our approach exhibits an even more pronounced advantage, surpassing the strongest baseline by 9.8% AUC. Moreover, in the radiology report generation downstream, our pre-trained model demonstrates remarkable adaptability, further attesting to the generality and clinical relevance of the learned representations. Our contributions are summarized below: \bullet We propose a CNN-ViT hybrid encoder that replaces ViT’s patch embedding with a 3D CNN and integrates multi-scale fusion, balancing efficiency and performance on 3D CT while retaining compatibility with pre-trained ViT priors. \bullet We introduce disease-level contrastive learning via learnable query tokens that dynamically extract and align disease-specific semantics from full reports—enabling fine-grained alignment without explicit report decomposition. \bullet We design a diagnosis-aware prompt strategy that uses real clinical descriptions and aggregated prototypes to bridge the training-inference gap and improve zero-shot diagnostic reliability. 2 Related Work 2.1 Medical Vision–Language Pre-training VLP leverages radiology reports as rich textual supervision to align images and text in a shared semantic space. Early work focused on 2D chest X-rays using global image–report contrastive learning ( 41 ; 2 ; 33 ; 5 ; 17 ; 6 ) . Recent efforts extend VLP to 3D CT ( 12 ; 4 ; 1 ; 22 ) , but still rely on coarse global alignment. This is problematic in 3D CT, where critical pathologies occupy only a small fraction of the volume, causing models to be dominated by non-diagnostic background. Recent studies have explored complementary strategies to address data and modeling limitations: leveraging LLMs to generate silver-standard labels for supervised pre-training ( 21 )