RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
Anas Khan, Mariam Husain, Kshitij Jadhav · 2026
Read RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture by Anas Khan, Mariam Husain, Kshitij Jadhav (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15891.txt.
Introduction
Anas Khan, Mariam Husain, and Kshitij Jadhav train a radiology encoder without text in a 2026 arXiv paper. Recent medical vision-language models guide visual learning with paired reports. That supervision is limited by how much paired image-text data exists. The authors ask whether robust radiology encoders can be learned without language. Their model is RadJEPA. It uses a joint embedding predictive architecture, per the title, and targets chest X-rays.
JEPA-style learning predicts representations of hidden image parts. It needs no captions at all. If it works, hospitals can train encoders on images they already hold. Paired reports are the bottleneck in medical AI; removing them widens access. A strong image-only encoder could then serve many downstream tasks, from disease detection to retrieval.
Catalog record
| CREATOR | Anas Khan, Mariam Husain, Kshitij Jadhav |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15891
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture Anas Khan, Mariam Husain, Kshitij Jadhav Abstract Recent advances in medical vision–language models guide the learning of visual representations; however, this form of supervision is constrained by the availability of paired image–text data, raising the question of whether robust radiology encoders can be learned without relying on language supervision. In this work, we introduce RadJEPA, a self-supervised framework built on a Joint Embedding Predictive Architecture that learns without language supervision. Pre-trained solely on unlabeled chest X-ray images, the model learns to predict latent representations of masked image regions. This predictive objective differs fundamentally from both image–text pre-training and DINO-style self-distillation: rather than aligning global representations across views or modalities, RadJEPA explicitly models latent-space prediction. We evaluate the learned encoder on disease classification, semantic segmentation, and report generation task. Across benchmarks, RadJEPA achieves performance exceeding state-of-the-art approaches including R ad -DINO. Code and pre-trained model weights are made publicly available on GitHub 1 1 1 https://github.com/aidelab-iitbombay/RadJEPA and Hugging Face 2 2 2 https://huggingface.co/AIDElab-IITBombay/RadJEPA . 1 Introduction Vision–language modeling has largely been driven by the use of textual supervision desai2021virtex ; radford2021learning . In biomedical imaging boecking2022making ; zhou2023advancing , these representations are often reused as fixed visual tokens for downstream reasoning, such as report generation, Image Classification, Segmentation. This paradigm implicitly assumes that textual descriptions provide a sufficiently complete and unbiased account of visual content dehghani2023scaling . Radiology reports are written to support clinical decision making and therefore emphasize selective findings, frequently omitting subtle variations, or absent observations. As a result, visual encoders trained to align images with text may inherit bias jones2024causal and fail to preserve the full semantic structure present in medical images. Beyond language supervision, recent self-supervised approaches such as DINO-style self-distillation have emerged as powerful alternatives for representation learning without labels oquab2023dinov2 . These methods learn invariances by aligning global representations across multiple augmented views of the same image using a teacher–student formulation. In the context of chest X-rays perez2025exploring , where fine-grained anatomical differences and subtle pathological cues are often diagnostically relevant, enforcing global agreement across views can lead to representations that prioritize appearance-level consistency over semantic completeness. Moreover, self-distillation methods dong2023maskclip emphasize consistency across pixel-space (view-centric) augmentations, rather than predicting semantic representations in latent space. Why Predictive Latent Modeling We argue that chest X-rays are fundamentally semantic rather than view-centric ccalli2021deep : clinical interpretation depends on global anatomical context, spatial relationships, and subtle deviations from normal structure. Consequently, learning representations directly in pixel space or enforcing view-level invariance may not be the most appropriate. Instead, we adopt a predictive latent modeling perspective assran2023self . Formally, given an image x , we partition it into visible regions x_{v} and masked regions x_{m} . An encoder f_{\theta} maps these regions into latent representations z_{v}=f_{\theta}(x_{v}) and z_{m}=f_{\theta}(x_{m}) . A predictor g_{\phi} is then trained to predict the latent representation of the masked regions from the latent representation of visible context by minimizing: \mathcal{L}=\left\|g_{\phi}(z_{v})-\text{stopgrad}(z_{m})\right\|^{2}. Our main contributions are as follows: • We show that language supervision and DINO-style self-supervised distillation have limitations in radiological representation learning. We introduce RadJEPA , a predictive self-supervised architecture that achieves performance comparable to or exceeding SoTA models trained on over 839k chest X-ray images. • We provide empirical evaluations of RadJEPA across three downstream tasks, classification, segmentation, and report generation. We benchmark against a diverse set of baselines, including larger-capacity architectures. 2 Related Work Contrastive learning has been a dominant paradigm for visual representation learning, relying on instance-level discrimination through positive and negative pairs oord2018representation ; chen2020simple ; he2020momentum . This formulation has been extended to multimodal settings, most notably in CLIP-style image-text models that align visual and textual embeddings in a shared latent space radford2021learning . While effective, contrastive objectives are known to induce representational invariances that may suppress fine-grained visual details and intra-class variability, particularly when supervision is incomplete or biased zhai2022lit ; liang2022mind . Such limitations are especially relevant in medical imaging, anatomical differences are critical for downstream analysis. Masked image modeling (MIM) and self-distillation approaches aim to reduce reliance on contrastive negatives and paired supervision. MIM-based methods learn contextualized representations by reconstructing masked regions or tokens bao2022beit ; he2022masked , demonstrating strong scalability and transferability. Complementary to this, self-distillation and clustering-based frameworks such as DINO enforce consistency across multiple augmented views without explicit negatives caron2021emerging . Recent large-scale variants (e.g., DINOv2) further show strong domain-transfer capabilities and applicability to dense prediction tasks oquab2023dinov2 . However, these approaches typically depend on pixel-level reconstruction or handcrafted augmentation strategies. Predictive learning frameworks have been proposed to address these limitations by learning representations through latent-space prediction rather than reconstruction or contrastive matching lecun2022path . Joint-Embedding Predictive Architectures (JEPA) formalize this idea by predicting abstract representations of masked or future regions, emphasizing semantic abstraction and representational stability assran2023jepa . Recent adaptations of this paradigm to neuroimaging, such as BrainJEPA, have demonstrated promising results in learning transferable and cognitively meaningful representations from brain signals without task-specific supervision dong2024brainjepa . Despite these advances, predictive architectures remain relatively underexplored in medical imaging zhou2019models ; tang2022self . 3 Methodology Figure 1: RadJEPA frozen encoder with trainable task-specific heads. 3.1 RadJEPA Pretraining Let x\in\mathbb{R}^{H\times W} denote an input chest X-ray. Two non-overlapping regions are sampled: a context region c and a target region t . A visual encoder f maps image regions to latent representations. A predictor network g projects the context embedding to the target embedding space: z_{c}=f(c),\qquad z_{t}=f(t),\qquad\hat{z}_{t}=g(z_{c}) RadJEPA is trained using a latent prediction objective. The objective operates entirely in representation space, without pixel reconstruction, contrastive negatives, or data augmentations. \mathcal{L}_{\text{JEPA}}=\mathbb{E}_{(c,t)}\left[\left\|\text{stopgrad}(z_{t})-\hat{z}_{t}\right\|_{2}^{2}\right] The parameters of the target encoder f^{\prime} are not optimized by gradient descent, but updated as an exponential moving average (EMA) of the online encoder f . \tau\in[0,1) is the momentum coefficient. \theta^{\prime}\leftarrow\tau\theta^{\prime}+(1-\tau)\theta Downstream Usage: After pretraining, the visual encoder f is frozen and reused across all downstream tasks. Task-specific heads or adapters are trained independently. 3.2 Disease Classification Given an image x , frozen features are extracted and a linear classifier W predicts class probabilities. Only the linear head is optimized. \displaystyle h \displaystyle=f(x),\qquad p=\text{softmax}(Wh), \displaystyle\mathcal{L}_{\text{cls}} \displaystyle=-\sum_{k=1}^{K}y_{k}\log p_{k} 3.3 Semantic Segmentation Multi-scale feature maps are extracted from the frozen encoder. A UperNet decoder \mathcal{D} aggregates hierarchical features to produce pixel-wise logits. Only decoder parameters are trained. \displaystyle\{h_{l}\}_{l=1}^{L} \displaystyle=f^{(l)}(x),\qquad S=\mathcal{D}(\{h_{l}\}) \displaystyle\mathcal{L}_{\text{seg}} \displaystyle=-\sum_{i,j}\sum_{c=1}^{C}y_{ij}^{c}\log\text{softmax}(S_{ij}^{c}) 3.4 Report Generation We follow a LLaVA-style multimodal formulation, extracting frozen visual embeddings v=f(x) from the input image. A lightweight two-layer projection adapter A maps visual features into the language embedding space. \tilde{v}=A(v)=v+\lambda W_{2}\sigma(W_{1}v), The projected embeddings are concatenated with an instruction prompt and provided as input to a language model \mathcal{M} . p(y_{t}\mid y_{<t},x)=\mathcal{M}(y_{t}\mid y_{<t},\tilde{v}) The report generation objective is the autoregressive negative log-likelihood. \mathcal{L}_{\text{txt}}=-\sum_{t=1}^{T}\log p(y_{t}\mid y_{<t},x) During training, the visual encoder f is frozen, while both the adapter parameters and the language model parameters are optimized jointly. Dataset View Patient cohort #Subjects #Images Frontal Lateral BRAX reis2022brax frontal, lateral institutional PACS 19,351 41,620 24,959 16,661 CheXpert chexpert frontal, lateral inpatient, outpatient 65,240 224,316 191,229 33,087 MIMIC-CXR johnson2019mimic frontal, lateral ICU 188,546 300,491 210,491 90,000 ChestX-ray14 wang2017chestx frontal not specified 32,717 112,120 112,120 0 PadChest Bustos_2020 frontal, lateral all available 67,000