Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture
Longxiang Zhang, Weilong DaiGuanghao Zhang, Hao JiangPipei HuangAlibaba Group {shengxiang.zlx, junmu.dwl, guanghao.zgh, aoshu.jh, pipei.hpp}@taobao.com · 2026
Read Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture by Longxiang Zhang, Weilong DaiGuanghao Zhang, Hao JiangPipei HuangAlibaba Group {shengxiang.zlx, junmu.dwl, guanghao.zgh, aoshu.jh, pipei.hpp}@taobao.com (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14448.txt.
Introduction
Think When Needed, from Zhang and colleagues in 2026, is a unified multimodal embedding system with adaptive reasoning. Recent methods add chain-of-thought to embeddings but cost heavily. For simple inputs, plain embeddings already work well. Think When Needed uses dual-LoRA adapters for reasoning and embedding on a shared frozen backbone. An adaptive think switch decides per input whether to use chain-of-thought, skipping needless reasoning. On MMEB-V2, it reaches state-of-the-art embedding quality with only 3 to 5 percent extra weights.
Catalog record
| CREATOR | Longxiang Zhang, Weilong DaiGuanghao Zhang, Hao JiangPipei HuangAlibaba Group {shengxiang.zlx, junmu.dwl, guanghao.zgh, aoshu.jh, pipei.hpp}@taobao.com |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66737 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14448
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Think When Needed: Adaptive Reasoning-Driven Multimodal Embeddings with a Dual-LoRA Architecture Longxiang Zhang, Weilong DaiGuanghao Zhang, Hao JiangPipei HuangAlibaba Group {shengxiang.zlx, junmu.dwl, guanghao.zgh, aoshu.jh, pipei.hpp}@taobao.com Abstract Multimodal large language models (MLLMs) have emerged as a powerful backbone for multimodal embeddings. Recent methods introduce chain-of-thought (CoT) reasoning into the embedding pipeline to improve retrieval quality, but remain costly in both model size and inference cost. They typically employ separate reasoner and embedder with substantial parameter overhead, and generate CoT indiscriminately for every input. However, we observe that for simple inputs, discriminative embeddings already perform well, and redundant reasoning can even mislead the model, degrading performance. To address these limitations, we propose T hink W hen N eeded (TWN), a unified multimodal embedding framework with adaptive reasoning. TWN introduces a dual-LoRA architecture that attaches reasoning and embedding adapters to a shared frozen backbone, detaching gradients at their interface to mitigate gradient conflicts introduced by joint optimization while keeping parameters close to a single model. Building on this, an adaptive think mechanism uses a self-supervised routing gate to decide per input whether to generate CoT, skipping unnecessary reasoning to reduce inference overhead and even improve retrieval quality. We further explore embedding-guided RL to optimize CoT quality beyond supervised training. On the 78 tasks of MMEB-V2, TWN achieves state-of-the-art embedding quality while being substantially more efficient than existing generative methods, requiring only 3–5% additional parameters relative to the backbone and up to 50% fewer reasoning tokens compared to the full generative mode. Our project page is at https://github.com/winterfell00/Think-When-Needed . † † footnotetext: * Equal contribution. † Project lead. ‡ Corresponding author. 1 Introduction Figure 1: (a) Comparison of multimodal embedding frameworks: (i) discriminative, (ii) unified generative, (iii) decoupled generative, and (iv) TWN (ours). (b) Performance on MMEB-V2. (c) Average reasoning tokens per input. Multimodal embeddings map heterogeneous inputs such as images, videos, visual documents, and text into a unified representation space, serving as a foundational component for cross-modal retrieval, multimodal retrieval-augmented generation [ 19 ] , and related applications. Pioneering studies adopted a CLIP-like dual-encoder architecture [ 28 , 12 , 40 ] and learned aligned cross-modal representations through large-scale image-text contrastive pre-training. However, such architectures struggle to handle interleaved multimodal inputs or to support complex, instruction-aware retrieval. With the rapid progress of multimodal large language models (MLLMs) [ 23 , 20 , 3 , 35 ] , the field has shifted toward adopting MLLMs as unified encoders [ 13 , 14 , 21 , 16 , 9 ] , which natively handle interleaved multimodal inputs and enable more complex retrieval scenarios through their strong understanding capabilities. Prevailing MLLM-based embedding methods feed the input through an MLLM and take the last-layer hidden state of the final token as the semantic representation, which we refer to as discriminative embeddings (Figure 1 (a-i)). This line of work treats the MLLM as a generic feature extractor and does not explicitly leverage its generation and reasoning abilities, and therefore tends to fall short for hard queries that demand fine-grained understanding and reasoning. Consequently, recent studies explicitly incorporate the reasoning ability of MLLMs into the embedding pipeline, yielding what we call generative embeddings [ 4 , 17 , 22 , 24 ] . Concretely, a reasoner first produces a chain-of-thought (CoT) [ 36 ] for the input, and an embedder then derives the semantic representation conditioned on both the original input and the generated CoT. These methods achieve noticeably better retrieval quality on hard queries than their discriminative counterparts. Despite this progress, current generative embedding methods remain limited in two key respects. Architecturally , decoupled methods (Figure 1 (a-iii); e.g., TTE [ 4 ] ) employ separate models for reasoning and embedding, incurring substantial parameter overhead. Unified methods (Figure 1 (a-ii); e.g., UME-R1 [ 17 ] ) share a single MLLM for both tasks; while parameter-efficient, the competing autoregressive and contrastive objectives may introduce gradient conflicts [ 39 ] that limit performance. In terms of inference, all existing methods generate CoT indiscriminately for every input, incurring substantial decoding overhead; yet for simple inputs, discriminative embeddings already perform well [ 17 ] , and unnecessary CoT can actively mislead the model, degrading retrieval quality. To address these limitations, we propose T hink W hen N eeded (TWN), a unified multimodal embedding framework with adaptive reasoning (Figure 1 (a-iv)). Our key ideas are: (1) a dual-LoRA architecture that attaches separate reasoning and embedding adapters to a shared frozen backbone, mitigating gradient conflicts while keeping parameters close to a single model; and (2) an adaptive think mechanism that selectively invokes CoT only when it benefits retrieval, skipping unnecessary reasoning for simple inputs. Our contributions are: • We propose a dual-LoRA architecture that attaches reasoning and embedding adapters to a shared frozen backbone with gradient detachment at their interface, combining the gradient isolation of decoupled methods with the parameter efficiency of unified methods while adding only a small fraction of backbone parameters. • We introduce an adaptive think mechanism with a routing gate that decides per input whether to generate CoT, skipping unnecessary reasoning to reduce inference cost and avoiding misleading CoT that can degrade retrieval quality on simple inputs. • We explore embedding-guided RL that exploits the parameter separation of dual-LoRA to freeze the embedding adapter as a stationary reward environment and introduces a global embedding cache for more discriminative reward signals. • On the 78 tasks of MMEB-V2, TWN achieves state-of-the-art embedding quality while being substantially more efficient, requiring only 3–5% additional parameters relative to the backbone and up to 50% fewer reasoning tokens compared to the full generative mode. 2 Method 2.1 Data Construction To provide high-quality CoT supervision for the reasoning adapter, we construct training data at scale through a generate-then-filter pipeline that spans diverse modalities (image, video, and visual document) and multiple datasets, and enforces alignment between reasoning traces and retrieval objectives through rigorous quality filtering. We first curate source data from multimodal tasks, following the data paradigm of VLM2Vec-V2 [ 25 ] : (1) image-based tasks from MMEB-train [ 14 ] , covering classification, question answering, retrieval, and grounding; (2) video-language instruction data from LLaVA-Hound [ 41 ] , including captioning, question answering, and retrieval; (3) visual document retrieval data from ViDoRe [ 7 ] and VisRAG [ 38 ] . For each retrieval pair (q,t^{+}) , we generate side-specific CoT traces independently for q and t^{+} . To prevent label leakage, the teacher model receives only one side at a time without access to its paired counterpart, producing structured traces in the format <think> … </think><answer> … </answer> , where <think> captures step-by-step analysis and <answer> provides a retrieval-oriented summary. The generation is guided by task-specific prompts, e.g., producing the most specific category label for classification and extracting key semantic elements for retrieval. We then filter the generated traces using a judge model with task-adaptive validation modes: strict verification (reasoning quality and answer matching) for tasks with well-defined answers, and hallucination-only verification for open-ended tasks. Samples that fail their respective criteria are removed so the model is trained only on CoT data that passes the filtering criteria. The final dataset comprises retrieval pairs (q_{n},t_{n}^{+}) with side-specific CoT traces c_{n} . For the teacher and judge model,