Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning
Julien Khlaut, Charles Corbière, Baptiste Callard, Amaury Prat, Leo Butsanets, Antoine Saporta, Théo Danielou, Leo Machado, Korentin Le Floch, Tom Boeken, Pierre Manceron, Corentin Dancette, [5pt] Equal contribution., [3pt] Raidium, Paris, 75014, France · 2026
Read Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning by Julien Khlaut, Charles Corbière, Baptiste Callard, Amaury Prat, Leo Butsanets, Antoine Saporta, Théo Danielou, Leo Machado, Korentin Le Floch, Tom Boeken, Pierre Manceron, Corentin Dancette, [5pt] Equal contribution., [3pt] Raidium, Paris, 75014, France (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24570.txt.
Introduction
Khlaut and colleagues teach a medical foundation model to align CT scans with reports organ by organ, not just globally. Vision language contrastive pretraining dominates three-dimensional medical imaging, but scans span dozens of organs and reports run long. CLIP style training compresses each side into one global token, and loses detail. Their method, ConQuer, adds localised alignments on top of the global one, one alignment per concept.
Reports split into concept specific sections, and learned cross attention queries pool the matching image features for each concept, with no segmentation masks or spatial supervision required. Contrastive learning then applies independently per concept. The liver's text aligns with liver features, rather than the whole scan. Concepts here are anatomical regions, one query per organ. The system, developed at Raidium in Paris and presented as Jolia in 2026, aims at foundation models that know which part of the scan a sentence describes.
Catalog record
| CREATOR | Julien Khlaut, Charles Corbière, Baptiste Callard, Amaury Prat, Leo Butsanets, Antoine Saporta, Théo Danielou, Leo Machado, Korentin Le Floch, Tom Boeken, Pierre Manceron, Corentin Dancette, [5pt] Equal contribution., [3pt] Raidium, Paris, 75014, France |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24570
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning Julien Khlaut, Charles Corbière, Baptiste Callard, Amaury Prat, Leo Butsanets, Antoine Saporta, Théo Danielou, Leo Machado, Korentin Le Floch, Tom Boeken, Pierre Manceron, Corentin Dancette, [5pt] Equal contribution., [3pt] Raidium, Paris, 75014, France Abstract Vision-language contrastive pretraining has become the dominant recipe for 3D medical foundation models, leveraging the large volumes of paired scans and reports produced in clinical practice. However, medical images usually span dozens of organs, and radiological reports are much longer than typical natural image captions and are composed of multiple structured sections. CLIP-style pretraining compresses this structure by encoding each modality into a single global token, at the risk of losing important details. We introduce ConQuer (Concept Queries), an image-text pretraining method that augments CLIP’s global alignment with a set of localized alignments, one per concept . ConQuer splits the report into concept-specific sections and learns cross-attention queries that pool the matching image features without using any segmentation mask or spatial supervision. Contrastive learning is then applied independently for each concept. Concepts can be any unit of semantic localization; here, they are anatomical regions, one query per organ or gross body region. As a byproduct, each query learns attention maps focused on its concept, providing built-in spatial interpretability. We use ConQuer to train Jolia, a 3D CT foundation model on chest and abdominal CT. Jolia consistently outperforms a CLIP baseline on findings classification, report generation, and cross-center transfer, and sets a new state of the art across multiple public benchmarks. Jolia’s weights are available here . 1 Introduction Foundation models 4 have transformed machine learning by demonstrating that large-scale pretraining on broad data, followed by lightweight adaptation, can outperform task-specific models trained from scratch. In medical imaging, this paradigm is particularly appealing: labeled data is scarce, annotation requires clinical expertise, and the diversity of downstream tasks (classification, segmentation, report generation, retrieval) makes it impractical to train dedicated models for each. Contrastive vision-language (VL) pretraining 22 sidesteps these constraints: it turns the image–report pairs already produced in clinical practice into free supervision, requiring no manual labels, and yields representations that transfer to linear probing, zero-shot classification, and cross-modal retrieval. A fundamental limitation of standard CLIP-style alignment, however, is that both the image and the report are compressed into single global vectors, while real inputs and their text typically decompose along structural axes that a single token cannot preserve 16 . In CT studies, a single scan may span dozens of organs, and a report typically describes findings organ by organ (e.g., “ Lungs: no nodule. Liver: 2cm cyst in segment VII. ”), producing text much longer than in typical CLIP pretraining. Recent fine-grained 3D CT methods recover this lost structure using segmentation masks 26 ; 32 ; 19 , but at the cost of organ-level supervision, which limits coverage to the organs the segmentation tool supports. We propose ConQuer (Concept Queries), a vision-language pretraining method that addresses these limitations by complementing the global CLIP objective with a parallel set of concept-level contrastive alignments. Any semantic axis along which one might want to localize the image–text alignment can serve. In this work, concepts are anatomical regions, and we use ConQuer to train Jolia, a 3D foundation model, on public abdominal and chest CT datasets. Our key contributions are: 1. ConQuer, a concept-level contrastive pretraining method without spatial supervision. We complement the global CLIP objective with a parallel set of concept-level contrastive losses, learned end-to-end from raw volume–report pairs. Each report is split by an LLM into concept-specific sections; on the image side, a small set of learnable cross-attention queries pools concept-specific features directly from the encoder, without requiring masks or spatial annotations. Contrastive alignment is then performed independently per concept, comparing the same concept across samples. 2. We use ConQuer to train the Jolia foundation model on 74,434 public chest and abdominal CT–report pairs from CT-RATE 11 , INSPECT 12 , and Merlin-Abd-CT 3 . Model weights will be released to the research community. 3. We benchmark Jolia against a CLIP baseline and competing models, and set a new state of the art on findings classification (linear probing and zero-shot, in- and out-of-distribution), cross-center transfer, and radiology report generation, while remaining competitive on image–text retrieval. We show that ConQuer generalizes across architectures and concept choices. Figure 1 : Overview of ConQuer. We decompose both the image (via learnable cross-attention queries) and the report (via concept-specific sections produced by an LLM) into a set of concept-level representations, and apply contrastive alignment independently for each concept in addition to the standard global \mathcal{L}_{\text{CLIP}}(z_{\texttt{[CLS]}},\,t) . 2 Related Work Foundation models in medical imaging. Vision-language pretraining has become a popular recipe for medical foundation models 34 ; 8 ; 25 , with paired images and clinical text routinely available from radiology workflows. In 3D CT specifically, three families have emerged: (1) global CLIP-style alignment between volume and report (CT-CLIP 10 , Merlin 3 on abdominal CT (whose dataset we use), Pillar-0 6 , COLIPRI 28 ); (2) self-supervised or multi-modal scaling without text-side locality (CT-FM 21 , SPECTRE 7 , Curia 9 , Curia-2 24 ); and (3) explicit anatomical decomposition via segmentation masks (fVLM 26 , TotalFM 32 , CT-GLIP 19 ). Local and region-based vision-language alignment. Several prior works recover the structure lost by global CLIP through finer-grained alignment: FILIP 33 and DenseCLIP 23 use token-wise or dense-pixel late interaction in natural images, while GLoRIA 13 and MGCA 29 extend the idea to chest X-rays at the word–patch and pathological-region level. ConQuer differs from these along three axes. (i) we produce one image and one text embedding per concept and apply an independent contrastive objective per concept, comparing the same concept across samples, whereas global CLIP-style methods (CT-CLIP, Merlin, Pillar-0, BiomedCLIP) compress each volume and report into a single token. (ii) unlike token-wise alignment (FILIP, GLoRIA, MGCA), where the alignment must emerge from the data, our queries impose an explicit concept-level decomposition that mirrors how reports already split their content along anatomical lines. (iii) unlike segmentation-based 3D approaches (fVLM, TotalFM, CT-GLIP), which depend on a pretrained 3D CT segmenter (e.g., RadSAM 17 ) to crop organs before alignment, our queries learn where to attend purely through the per-concept contrastive loss, with no spatial supervision. Learnable-query visual representations. Compressing high-resolution visual inputs into a small set of learnable tokens via cross-attention is a well-established mechanism: DETR 5 uses object queries for end-to-end detection, the Perceiver 14 uses latent queries as a general bottleneck for arbitrary input modalities, and BLIP-2’s Q-Former 18 uses 32 learnable query tokens to bridge a frozen image encoder and a language model. Our cross-attention queries follow the same design, but anchor each query to a specific concept (in our CT setting, an anatomical region) and tie it to a per-concept contrastive objective rather than to a downstream detection or generation task. 3 Method ConQuer is a vision-language pretraining method that aligns volumes and reports at the level of concepts , rather than at the global level alone. A concept is a single semantic unit along which we localize the alignment; in this paper, concepts are the K entries of an anatomical taxonomy, typically a single organ (liver, spleen,