QwenCLIP: Boosting Medical Vision-Language Pretraining via LLM Embeddings and Prompt tuning

Xiaoyang Wei, Camille Kurtz, Florence Cloppet · 2025

Read QwenCLIP: Boosting Medical Vision-Language Pretraining via LLM Embeddings and Prompt tuning by Xiaoyang Wei, Camille Kurtz, Florence Cloppet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13876.txt.

Introduction

Xiaoyang Wei, Camille Kurtz, and Florence Cloppet address a size limit that hampers medical image AI. The popular CLIP framework pairs images with text, but its text reader caps out at 77 tokens, far too short for full radiology reports. Even medical text encoders stop near 512 tokens and grasp meaning only shallowly.

Their system, QwenCLIP, swaps in a large language model embedding module with a much longer input window and adds learnable prompts to sharpen image-text matching. The change lets the model take in long clinical reports in full. The authors report better alignment and stronger results on radiology benchmarks, and they release the code publicly.

Catalog record

CREATORXiaoyang Wei, Camille Kurtz, Florence Cloppet
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13876

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

QwenCLIP: Boosting Medical Vision-Language Pretraining via LLM Embeddings and Prompt tuning Xiaoyang Wei, Camille Kurtz, Florence Cloppet Abstract Contrastive Language–Image Pretraining (CLIP) has demonstrated strong generalization for vision-language tasks in computer vision and medical domains, yet its text encoder accepts only up to 77 tokens, which limits its ability to represent long and information-rich radiology reports. Recent adaptations using domain-specific encoders, such as PubMedBERT or ClinicalBERT, mitigate this issue by leveraging medical corpora, but remain constrained by their limited input length (typically 512 tokens) and relatively shallow semantic understanding. To address these limitations, we propose QwenCLIP, a vision-language framework that replaces CLIP’s text encoder with a large language model (LLM)–based embedding module (e.g., Qwen3-Embedding) and introduces learnable prompts to enhance cross-modal alignment. By leveraging the extended context window and richer representations of LLMs, QwenCLIP captures comprehensive medical semantics from long-form clinical text, substantially improving medical image–text alignment and downstream performance on radiology benchmarks. Our code is publicly available at https://github.com/Wxy-24/QwenCLIP . † † address: Laboratoire d’Informatique Paris Descartes (LIPADE), Université Paris Cité (France) 1 Introduction The advent of Contrastive Language–Image Pretraining (CLIP) [ 9 ] has revolutionized medical visual representation learning, enabling models to acquire rich multimodal understanding through self-supervised learning rather than relying on exhaustive manual annotation. Instead of using expensive and time-consuming expert annotations for medical images, researchers are increasingly leveraging abundant image–text pairs readily available in clinical and public data sources. These pairs typically originate from two major domains: (1) radiological images with corresponding diagnostic reports and clinical notes routinely stored in Picture Archiving and Communication Systems (PACS) [ 5 ] , and (2) medical figures and captions extracted from publicly available biomedical literature [ 10 ] . Such resources offer scalable and diverse supervision signals for aligning visual and textual representations in the medical domain [ 12 ] . Figure 1: Text length distribution for common medical image-text pair datasets. Although CLIP has achieved remarkable success by utilizing large-scale, freely available image–text pairs, it still presents several non-negligible limitations when directly applied to medical data. One major constraint lies in its text encoder, which accepts no more than 77 tokens—a design choice suitable for short, web-scraped captions (e.g., “a cat sitting on a sofa”), but clearly insufficient for long and information-dense medical texts. Radiology reports, for instance, often exceed several hundred tokens, containing detailed descriptions of anatomical structures, imaging findings, and diagnostic impressions. When fed into CLIP, such text must be truncated, resulting in inevitable information loss and degraded alignment with the corresponding medical image. To better capture domain-specific semantics, several studies have proposed replacing CLIP’s text encoder with medical language models such as PubMedBERT [ 2 ] or ClinicalBERT [ 3 ] , which are pretrained on large-scale biomedical corpora. These models indeed improve text understanding by incorporating medical terminology and contextual nuances. However, BERT-based encoders inherently support a limited context window (up to 512 tokens), which remains inadequate for processing long-form clinical narratives—especially multi-section radiology reports or comprehensive case descriptions (as illustrated in Fig. 1 ). Thus, while domain-specific BERT variants alleviate part of the problem, they cannot fully address the context length bottleneck. In contrast, large language models (LLMs) have recently demonstrated extraordinary capabilities in handling long textual contexts and capturing rich semantic relationships across sentences and paragraphs [ 7 ] . Furthermore, LLMs are typically pretrained on vast, diverse corpora spanning the entire internet, allowing them to benefit from scaling laws and acquire a broad base of intrinsic knowledge. This often enables them to outperform BERT-like encoders that are trained on more limited domain-specific data. For example, Gemini 2.5 Pro can process up to 1 million tokens, such technological advancement can effectively eliminate the concern of the input-length limitations faced by BERT and the original CLIP encoder. This ability is particularly advantageous for modeling detailed medical reports, where long radiology descriptions frequently exceed conventional context windows and risk truncation. Consequently, employing an LLM-based text encoder provides a promising avenue for enhancing medical vision–language alignment. Recent work such as LLM2CLIP [ 4 ] has explored integrating general-purpose LLMs (e.g., LLaMA-based models [ 1 ] ) into the CLIP framework, demonstrating the potential of large language models but introduce significant complexities: they require extensive two-stage training pipelines where generative LLMs must first be converted into discriminative encoders through caption contrastive fine-tuning. This not only increases computational overhead but also represents a fundamental architectural mismatch—adapting text generation models for embedding tasks. In contrast, we propose QwenCLIP, which fundamentally rethinks this integration strategy. Rather than repurposing generative LLMs through complex adaptation pipelines, it directly employs Qwen3-embedding [ 13 ] —a dedicated embedding model specifically designed for representation learning. This eliminates the need for the cumbersome pre-processing stages required by methods like LLM2CLIP while providing native support for long contexts. Furthermore, recognizing the sensitivity of LLMs to input phrasing and prompts, QwenCLIP introduces a hybrid prompt tuning strategy that enables the model to autonomously learn task-adaptive prompts for effective cross-modal alignment. Together, these innovations allow QwenCLIP to bridge the gap between long-text medical understanding and visual representation learning, providing an efficient, scalable, and semantically rich solution for medical vision–language modeling. Figure 2: Overview of the QwenCLIP framework, where instruction-tuned soft prompts guide the frozen LLM and learnable image encoder to align multimodal representations 2 Proposed framework: QwenCLIP Given a dataset of paired medical images and captions, our QwenCLIP framework aims to learn semantically aligned visual embeddings for downstreams tasks like image retrieval. The method extends the original CLIP architecture by (1) replacing its text encoder with the Qwen3-embedding-8B model [ 13 ] , which is a high-capacity LLM optimized for dense text representations, and (2) introducing a hybrid prompt tuning strategy to better capture medical contextual information (as illustrated in Fig. 2 ). 2.1 Hybrid prompt tuning for LLM-based text encoder Traditional CLIP often employs a fixed template (e.g., “a photo of a \ldots ”) for zero-shot classification. However, such handcrafted prompts tend to be inadequate for medical contexts where textual descriptions are domain-specific and context-dependent. Inspired by CoOp [ 14 ] , we design a hybrid prompting strategy that combines static explicit prompts with a learnable contextual soft prompt . Specifically, we prepend the input caption with explicit prefixes such as “Instruct:” and “query:” to provide structural cues to the LLM about the intended task (instructional vs. retrieval-oriented semantics). In addition to these static tokens, we introduce a learnable soft prompt vector P_{s} designed with the following form:, P_{s}=[V]_{1}[V]_{2}\dots[V]_{K}. where each [V]_{k}\ (k\in\{1,\dots,K\}) is a learnable vector with the same dimension as word embeddings, and K is a hyperparameter corresponding to the number of context tokens which depend on the initialization text. Specifically, we initialized P_{s} using the following phrase with K=15 in this work: “Create a dense embedding that represents the medical meaning of this text for image retrieval.” During training, P_{s} is optimized jointly with the model parameters, allowing the text encoder to dynamically refine the contextualized embedding space while preserving the linguistic priors of the pretrained Qwen3-embedding model. Formally, given a caption T , we construct the final textual input as: T^{\prime}=[\text{Instruct: $P_{s}$, query:}T]. where “Instruct:” and “query:” refer to the static explicit prompt during training. Finally, the Qwen3-embedding model encodes T^{\prime} to

Read the complete text