Unified Medical Image Tokenizer for Autoregressive Synthesis and Understanding

Chenglong Ma † 1,2, Yuanfeng Ji † 3, Jin Ye 4, Zilong Li 1, Chenhui Wang 1, Junzhi Ning 4, Wei Li 4, Lihao Liu 4, Qiushan Guo 5, Tianbin Li 4, Junjun He ∗ 2,4, Hongming Shan ∗ 1, 1 Fudan University 2 Shanghai Innovation Institute 3 Stanford University, 4 Shanghai AI Laboratory 5 ByteDance Seed · 2025

Read Unified Medical Image Tokenizer for Autoregressive Synthesis and Understanding by Chenglong Ma † 1,2, Yuanfeng Ji † 3, Jin Ye 4, Zilong Li 1, Chenhui Wang 1, Junzhi Ning 4, Wei Li 4, Lihao Liu 4, Qiushan Guo 5, Tianbin Li 4, Junjun He ∗ 2,4, Hongming Shan ∗ 1, 1 Fudan University 2 Shanghai Innovation Institute 3 Stanford University, 4 Shanghai AI Laboratory 5 ByteDance Seed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19225.txt.

Introduction

Models that reason step by step over images must first chop each image into tokens, and medicine lacks a tokenizer that keeps fine anatomy while carrying clinical meaning. Chenglong Ma and a team of co-authors build one in this 2025 paper. Mixing the two goals in one training run, they note, makes the goals fight and wastes the many medical images that have no captions.

Their fix trains in two stages, using a visual representation as a bridge. First the tokenizer learns from more than thirty-three million unpaired images across nine scan types, securing faithful reconstruction. Then two million captioned pairs inject detailed text meaning without wrecking the first stage. The result, MedITok, led on over thirty benchmarks spanning the nine modalities, the authors report.

Catalog record

CREATORChenglong Ma † 1,2, Yuanfeng Ji † 3, Jin Ye 4, Zilong Li 1, Chenhui Wang 1, Junzhi Ning 4, Wei Li 4, Lihao Liu 4, Qiushan Guo 5, Tianbin Li 4, Junjun He ∗ 2,4, Hongming Shan ∗ 1, 1 Fudan University 2 Shanghai Innovation Institute 3 Stanford University, 4 Shanghai AI Laboratory 5 ByteDance Seed
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19225

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unified Medical Image Tokenizer for Autoregressive Synthesis and Understanding Chenglong Ma † 1,2, Yuanfeng Ji † 3, Jin Ye 4, Zilong Li 1, Chenhui Wang 1, Junzhi Ning 4, Wei Li 4, Lihao Liu 4, Qiushan Guo 5, Tianbin Li 4, Junjun He ∗ 2,4, Hongming Shan ∗ 1, 1 Fudan University 2 Shanghai Innovation Institute 3 Stanford University, 4 Shanghai AI Laboratory 5 ByteDance Seed Abstract Autoregressive modeling has driven major advances in multimodal AI, yet its application to medical imaging remains constrained by the absence of a unified image tokenizer that simultaneously preserves fine-grained anatomical structures and rich clinical semantics across heterogeneous modalities. Existing approaches jointly optimize image reconstruction and textual semantic objectives, relying on large-scale image-caption pairs and are prone to gradient interference. This is ill-suited for the medical domain where paired data are scarce and abundant unpaired images remain unexploited. This work identifies these issues in building unified medical image tokenizers, and introduces a principled two-stage training framework using visual representation as a bridge to address them. The propose visual representation alignment stage enables the utilization of large-scale unpaired medical images to ensure reconstruction fidelity and establish foundational semantics, alleviating the interference and better preparing for the second stage where fine-grained textual semantics are injected using image-text pairs. The resulting tokenizer, MedITok, is trained on over 33 million medical images spanning 9 modalities and 2 million image-text pairs. MedITok achieves state-of-the-art performance on 30+ benchmarks spanning 9 imaging modalities and 4 task families. It further enables autoregressive modeling for diagnostic and generative applications, serving as a scalable component for future multimodal models with unified synthesis and understanding capabilities in the medical domain. Project page: https://github.com/Masaaki-75/meditok † † footnotetext: † Equal contribution. ∗ Co-correspondence. 1 Introduction The rapid evolution of advanced autoregressive (AR) models, such as GPT-4o [ 219 ] , has revolutionized multimodal learning. These models excel at generating and understanding text, image, and audio data via unified processing of token-based representations. In medical imaging, AR models begin to demonstrate similar promise, powering report generation [ 284 ] , tumor segmentation [ 59 ] , counterfactual synthesis [ 188 ] , and diagnostic visual question answering (VQA) [ 165 ] . By translating complex biomedical image patterns into token sequences, these models can synthesize realistic images and interpret clinical cues ( e.g . , ground-glass opacities on chest computed tomography, pleural effusions on X-rays) in the images, with the potential to streamline workflows and improve patient outcomes. A critical ingredient in building a powerful AR model is the image tokenizer , which translates an input image to a sequence of discrete tokens suitable for AR modeling. Existing approaches can be divided into two categories. (1) Synthesis-oriented tokenizers optimized for pixel-level reconstruction 1 1 1 In this paper, “reconstruction” refers to autoencoding reconstruction: decoding an input image from its latent representation. , e.g . , VQGAN [ 82 ] . These tokenizers precisely capture low-level structure in the image that is vital to image compression [ 295 , 305 ] and generation [ 333 , 282 , 321 , 318 ] . However, they do not explicitly encode discriminative features and are therefore not suitable for interpreting the concepts and objects embedded in the image. (2) Understanding-driven tokenizers trained with discriminative objectives, e.g . , CLIP [ 244 ] . These tokenizers excel at capturing high-level textual semantics, making them indispensable for visual comprehension, but they fail to accurately retain spatial structures and textures in the image. Motivation. Image tokens that embed only one side of this structure-semantic spectrum will offload the representation learning burden onto downstream AR models, which often incurs heavy pre-training costs and can still leave either side under-utilized [ 304 , 60 ] . These limitations are especially acute in the medical domain, where clinical tasks typically demand both precise visual structures and clinical semantics. However, current medical image tokenizers tend to specialize in one single aspect [ 186 , 325 , 328 , 121 ] , which lacks a unified, information-rich token space and thereby limits the potential of downstream medical AR models for accurate and data-efficient diagnosis. Our goal is to democratize a foundation unified tokenizer for medical images. Nonetheless, training a unified tokenizer for medical images poses unique challenges: (1) a naïve joint optimization of image reconstruction and textual semantic objectives may introduce mutual interference and suboptimal performance [ 312 , 242 ] ; and (2) paired image-caption data for training is much more scarce in the medical domain, compared to the abundant unlabeled images. To addresses these issues, we propose a novel two-stage training framework. Instead of directly coupling the visual reconstruction and textual semantic, it involves a visual representation alignment stage as a bridge to first establish basic semantic awareness with reconstruction capabilities, followed by the textual semantic alignment stage for learning finer-grained semantic information. This framework leads to our model: MedITok, the first unified medical image tokenizer. MedITok encodes both low-level structural information, supporting faithful image reconstruction and realistic image synthesis, and high-level semantics, enabling multimodal medical image comprehension, serving as a foundation for diverse community use. Specifically, the first training stage trains MedITok on pure medical images, optimizing for reconstruction fidelity with a visual semantic constraint on the latent space. Then, the textual semantic alignment stage tunes MedITok on image-caption pairs, enhancing semantic richness by aligning image tokens to fine-grained textual embeddings of detailed captions. This approach allows MedITok to effectively encode structural and semantic information while exploiting both unpaired medical images and image-text pairs at scale , making a unified latent space to develop powerful AR models for diverse tasks. To achieve this, we meticulously collect a large-scale dataset comprising over 30 million medical images and 2 million image-caption pairs from multiple public sources, with broad coverage of imaging modalities, anatomies, and pathologies. This collection ensures that MedITok learns robust representations for medical images across diverse clinical contexts. Contributions. (1) We propose a novel training framework for developing a unified image tokenizer, which effectively scales up with medical image data and progressively builds a unified latent space. (2) We introduce MedITok, the first Medical Image Tokenizer that unifies the encoding of low-level structural details and high-level clinical semantics. (3) Extensive experimental results on over 30 datasets, spanning 9 imaging modalities, across 4 different task families, showcase MedITok’s state-of-the-art performance over existing approaches and broad applicability to diverse medical tasks. (4) Model and code will be open-source. Data accesses are provided respecting all original licenses. 2 Related Work AR Models in Medical Image Tasks. AR models have shown impressive scalability and generalizability in general vision-language processing. In medical domain, these models have been extended to a variety of tasks: image captioning and VQA for interpreting scans and assist diagnosis [ 165 , 203 , 57 ] , lesion segmentation model across different imaging modalities [ 59 ] , medical image synthesis for counterfactual analysis [ 188 ] and modality transfer [ 246 ] , etc . More recently, HealthGPT [ 172 ] , empowered by the VQGAN as the image tokenizer, further unifies both medical image synthesis and comprehension capabilities within an AR framework for broader applications. However, these methods typically general-domain tokenizers pre-trained on natural

Read the complete text