SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignment

Weiren Zhao, Yi Dong, Cheng Chen · 2026

Read SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignment by Weiren Zhao, Yi Dong, Cheng Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08724.txt.

Introduction

Weiren Zhao, Yi Dong, and Cheng Chen unite medical image understanding with medical image generation in one model, SynerMedGen. Earlier unified medical models treated the two jobs as separate goals with no real synergy. The authors ask what kind of understanding actually helps generation, and build training around that answer with three understanding tasks aligned to generation plus a two stage schedule.

Understanding training alone already yields strong zero shot results across twenty two synthesis tasks and fresh datasets. Adding generation training then beats both specialized synthesizers and recent unified rivals. A large released dataset pairs a million synthesis samples with two million derived understanding cases. Medical AI gains a working recipe for synergy, not just coexistence.

Catalog record

CREATORWeiren Zhao, Yi Dong, Cheng Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08724

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignment Weiren Zhao, Yi Dong, Cheng Chen Abstract Unifying multimodal understanding and generation is a compelling frontier that is beginning to emerge in the medical field. However, the limited existing unified medical models typically treat understanding and generation as disjoint objectives, lacking a meaningful functional synergy. In this work, we identify and address a critical question in unified medical modeling: what form of “understanding” truly benefits generation. We present SynerMedGen, a unified framework built on the proposed principle of generation-aligned understanding, which synergizes understanding objectives with generation tasks via task alignment. SynerMedGen introduces three generation-aligned understanding tasks and a two-stage training strategy that transfers generation-beneficial representations learned during understanding training to medical image synthesis. Remarkably, even with understanding training alone, our SynerMedGen achieves strong zero-shot performance across 22 medical image synthesis tasks and demonstrates robust generalization to unseen datasets. When combined with generation training, SynerMedGen consistently outperforms state-of-the-art specialized medical image synthesis models as well as recent unified medical models. We also release a large-scale dataset named SynerMed consisting of 1M paired synthesis samples and 2M generation-derived understanding instances to support further research on understanding-generation synergy. Our project can be accessed at https://github.com/Mhilab/SynerMedGen . Figure 1 : Overview of the comparison of the proposed generation-aligned understanding supervision and tradition understanding supervision. The right panel shows a comparison of SSIM values on the synthetic tasks under different understanding settings, as well as a comparison of accuracies on generation-aligned understanding tasks. 1 Introduction Recent years have witnessed rapid and parallel advances in multimodal understanding and visual generation, driven respectively by large vision-language models ( 2 ; 11 ; 25 ; 21 ) and diffusion-based generative models ( 33 ; 50 ; 36 ) . More recently, this progress has begun to converge in multimodal large language models (MLLMs) that unify understanding and generation within a single framework, enabling models to both interpret complex visual semantics and synthesize visual content ( 40 ; 48 ; 46 ; 47 ; 26 ) . Despite this momentum, such unified paradigms remain largely underexplored in the medical domain, where they are especially compelling as clinical imaging workflows require interpreting and transforming over heterogeneous modalities such as CT, multi-sequence MRI, PET to support diagnosis, treatment planning, and monitoring ( 20 ; 44 ) . Very recent efforts such as HealthGPT ( 23 ) and UniMedVL ( 31 ) have begun to explore unified medical MLLMs that couple medical understanding with image generation. HealthGPT supports both capabilities via separate task-specific adapters, while UniMedVL follows a progressive learning curriculum for understanding and generation. Despite this progress, a core question remains unresolved: what kind of “understanding” is actually beneficial for medical image generation? In exisiting frameworks, the understanding branch is typically trained with proxy objectives that are only weakly related to the generation tasks. As illustrated in Figure 1 , understanding tasks that emphasize global semantics can be insufficient for medical image generation tasks such as cross-modality synthesis, which requires retaining fine-grained anatomical/pathological content, respecting target-modality constraints, and maintaining pixel-level correspondence. This yields task misalignment that a model may become better at answering recognition-style questions, yet still fail to provide generation-beneficial representations ( 8 ; 12 ) . To address this limitation, we propose a simple but general principle: align understanding with generation by deriving understanding tasks from the generation dataset itself . We refer to this as a generation-aligned understanding, because the understanding tasks are co-defined with the generation objectives. In medical image generation, cross-modality image synthesis represents one of the most prevalent and important generation tasks, yet remains highly challenging, as it requires preserving patient-specific anatomy and pathology while modifying only modality-dependent appearance ( 35 ; 15 ) . Given its representativeness and difficulty, we focus on cross-modality image synthesis as a testbed for validating our design principle of medical understanding and generation model. For cross-modality medical image synthesis, the understanding tasks should facilitate (i) retain invariant clinical content to enable faithful synthesis, (ii) represent modality as an explicit controllable factor rather than an entangled cue, and (iii) encode the direction and constraints of the synthesis so that generation follows the intended modality change rather than an ambiguous appearance shift ( 16 ; 4 ; 53 ) . These criteria are employed to guide the design of our generation-aligned understanding tasks. Following these design principles, we propose SynerMedGen, a unified framework which synergize medical multimodal understanding with generation via generation-aligned understanding task design. Our SynerMedGen constructs a minimal set of synthesis aligned vision-language tasks directly from paired synthesis data: (1) Conditional Target Selection enforces one-to-one paired correspondence under an explicit target-modality request; (2) Modality Identification makes modality an explicit, learnable control factor; and (3) Transformation Instruction Alignment grounds paired visual differences in text to specify what should change (appearance/contrast) and what must remain invariant (structures/lesions), disambiguating synthesis direction. We employ a two-stage training strategy. First, the model learns these aligned tasks to obtain a generation-beneficial representation. Second, we optimize latent-space conditional synthesis on the same paired data, effectively transferring the learned multimodal priors into the generation branch. Our main contributions are highlighted as follows: • We identify a fundamental misalignment between understanding and generation in existing unified medical MLLMs, and propose SynerMedGen, that explicitly aligns understanding task design with generation objectives to synergize understanding with generation. • We design three generation-aligned understanding tasks, that effectively learn representations beneficial for medical image synthesis. • We conduct extensive evaluations of SynerMedGen against state-of-the-arts across 22 tasks, as well as on zero-shot performance, and generalization to unseen datasets, demonstrating strong improvements. • We release SynerMed, a large-scale dataset containing 1M paired samples and 2M generation-aligned understanding questions. 2 Related Work 2.1 Medical Multimodal Large Language Models Early medical MLLMs typically couple a medical vision encoder with a general-purpose LLM through a projection module, enabling strong performance on comprehension-oriented tasks such as VQA and report generation ( 20 ; 41 ; 49 ; 30 ; 52 ) . However, most of these systems are not designed to perform medical image synthesis or modality translation within the same model, and generation is commonly delegated to separate diffusion or synthesis pipelines ( 20 ; 51 ; 29 ; 7 ) . A parallel line of work improves medical MLLMs via data and instruction engineering, e.g., scaling medical vision–language pairs, cleaning noisy alignments, and instruction tuning ( 24 ; 3 ) . More recently, efforts such as HealthGPT explore unified training and lightweight adaptation to bridge understanding and generation ( 23 ) . Nevertheless, a central challenge remains: the understanding supervision is often not aligned with the information needed for medical images synthesis (e.g., slice-level correspondence, anatomy preservation, and modality-conditioned appearance changes). Our work addresses this limitation by deriving generation-aligned understanding tasks directly from paired synthesis data, so that the learned representation is explicitly optimized to support the generation task. Figure 2 : SynerMedGen overview. From 1M paired samples, we construct 2M generation-aligned understanding instances for three tasks: Conditional Target Selection (CTS), Modality Identification (MI), and Transformation Instruction Alignment (TIA). Stage I (GAU) learns a synthesis-sufficient representation; Stage II (UCG) performs flow matching in VAE latent space. 2.2 Unified Multimodal Understanding and Generation Unified multimodal models

Read the complete text