Reasoning-Driven Multimodal LLMs for Domain Generalization
Zhipeng Xu, Zilong Wang, Xinyang Jiang, Dongsheng Li, De Cheng, Nannan Wang · 2026
Read Reasoning-Driven Multimodal LLMs for Domain Generalization by Zhipeng Xu, Zilong Wang, Xinyang Jiang, Dongsheng Li, De Cheng, Nannan Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-23777.txt.
Introduction
Zhipeng Xu and five co-authors study domain generalization, the task of making image classifiers work on new visual styles they were never trained on. The 2026 paper tests whether step-by-step reasoning from multimodal language models can help, rather than relying only on visual features that stay fixed across styles.
The team built DomainBed-Reasoning, which pairs each sample with a reasoning chain leading to its class. They find two hurdles: training on reasoning chains is harder than training on labels alone, and mismatches between taught chains and model chains trade richness against ease of training. Their system, RD-MLDG, adds a direct classification path and a self-labeling rule that keeps chains rich while easing that mismatch. Tests on four standard sets reach the best reported scores.
Catalog record
| CREATOR | Zhipeng Xu, Zilong Wang, Xinyang Jiang, Dongsheng Li, De Cheng, Nannan Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-23777
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reasoning-Driven Multimodal LLMs for Domain Generalization Zhipeng Xu, Zilong Wang, Xinyang Jiang, Dongsheng Li, De Cheng, Nannan Wang Abstract This paper addresses the domain generalization (DG) problem in deep learning. While most DG methods focus on enforcing visual feature invariance, we leverage the reasoning capability of multimodal large language models (MLLMs) and explore the potential of constructing reasoning chains that derives image categories to achieve more robust predictions under domain shift. To this end, we systematically study the role of reasoning in DG using DomainBed-Reasoning, a newly constructed extension of DomainBed dataset, in which each sample is paired with class-relevant reasoning chains. Our analysis reveals two key challenges: (i) fine-tuning MLLMs with reasoning chains for classification is more challenging than direct label supervision, since the model must optimize complex reasoning sequences before label prediction; and (ii) mismatches in reasoning patterns between supervision signals and fine-tuned MLLMs lead to a trade-off between semantic richness (informative but harder to optimize) and optimization efficiency (easier to optimize but less informative). To address these issues, we propose RD-MLDG (Reasoning-Driven Multimodal LLM for Domain Generalization), a framework with two components: (i) MTCT (Multi-Task Cross-Training), which introduces an additional direct classification pathway to guide reasoning supervision; and (ii) SARR (Self-Aligned Reasoning Regularization), which preserves the semantic richness of reasoning chains while mitigating reasoning-pattern mismatches via iterative self-labeling. Experiments on standard DomainBed datasets (PACS, VLCS, OfficeHome, TerraInc) demonstrate that RD-MLDG achieves state-of-the-art performances, highlighting reasoning as a promising complementary signal for robust out-of-domain generalization. 1 1 footnotetext: 🖂 Corresponding author: De Cheng ([email protected]) 2 2 footnotetext: * Equal contribution: Zhipeng Xu, Zilong Wang and Xinyang Jiang contributed equally to this work. 3 3 footnotetext: Work done during an internship at Microsoft Research Asia. 1 Introduction In real-world scenarios, deep learning models are often required to generalize reliably to unseen distributions. However, due to domain shift, their performance often degrades substantially when deployed in new environments. To address this issue, Domain Generalization (DG) has emerged as a key research area, aiming to learn representations and prediction functions that transfer well to unseen domains using only source-domain data. Existing DG approaches generally fall into four categories: invariant representation learning, data augmentation, regularization, and meta-learning. While effective, these methods primarily focus on feature-level invariance and often fail to capture higher-level cross-domain commonalities, limiting their ability to generalize in complex scenarios. Recently, multi-modal large language models (MLLMs) have made significant progress and demonstrate strong reasoning capabilities. This capability offers a new opportunity for DG: rather than relying solely on invariant feature representations, one could exploit reasoning chains to bridge domain gaps and achieve more robust generalization. By producing structured, class-relevant reasoning chains, we can leverage MLLMs to explicitly decompose a classification task into interpretable and domain invariant steps with strong transferability across domains (Fig. 1 ). To systematically study the role of reasoning in DG, we leverage GPT-4o to construct DomainBed-Reasoning, an extension of the widely used DomainBed benchmark, where each sample is paired with class-relevant reasoning chains. Specifically, we investigate whether an MLLM fine-tuned with DomainBed-Reasoning can leverage generated reasoning chains to improve domain-generalized predictions. Figure 1: Illustrative examples of the printer in OfficeHome (Art Painting, Clipart, Product, Real World). Although the visual appearance differs substantially, the highlighted green segments of the reasoning chains remain highly consistent, capturing class-relevant cues that are likely to generalize well across domains. However, our empirical study shows that simply applying reasoning-chain supervision to classification tasks remains challenging. Our analysis reveals two key issues. First, reasoning-chain supervision is inherently difficult to optimize. In contrast to direct label prediction, the model must first fit complex reasoning chains before producing the final label. Second, reasoning chains generated by commercial models and those produced by fine-tuned MLLMs often exhibit distinct reasoning patterns. Commercial-model chains usually contain detailed contextual information that can assist classification, such as descriptions of background, object attributes, or scene conditions. While these chains are semantically rich and potentially informative, they are difficult for the model to optimize. In contrast, self-generated chains from the fine-tuned MLLM tend to be simpler in logic and more label-focused, which makes them easier to fit but less informative for classification. These observations indicate that straightforward reasoning supervision is insufficient and instead call for a more principled integration strategy. This motivates us to design RD-MLDG (Reasoning-Driven Multimodal LLM for Domain Generalization), the first DG framework that explicitly incorporates class-relevant reasoning chains to enhance out-of-domain performance. RD-MLDG consists of two key components: (i) MTCT (Multi-Task Cross-Training), which jointly optimizes a direct classification pathway and a reasoning-augmented pathway. The direct pathway focuses on standard label prediction, offering a simple and stable training signal. This signal guides the reasoning pathway by preventing it from fitting to overly complex or inconsistent reasoning chains, and instead keeps the supervision focused on the core classification objective; and (ii) Self-Aligned Reasoning Regularization (SARR), which retains informative reasoning chains while mitigating reasoning-pattern mismatches through self-generated supervision. Together, these components enable RD-MLDG to provide a principled way to integrate reasoning into DG for robust generalization. In summary, our work makes the following contributions: • We construct DomainBed-Reasoning, an extension of the DomainBed dataset where each sample is paired with class-relevant reasoning chains, enabling systematic evaluation of both classification accuracy and reasoning consistency under domain shift. • We propose RD-MLDG, the first DG framework that explicitly integrates reasoning by addressing two key challenges: (i) the difficulty of optimizing reasoning-chain supervision, and (ii) mismatches in reasoning patterns between supervision signals and fine-tuned MLLMs. • We empirically show that RD-MLDG achieves state-of-the-art performance on standard DomainBed benchmarks (PACS, VLCS, OfficeHome, TerraInc), demonstrating the effectiveness of reasoning-based approaches for DG. 2 Related work Domain generalization (DG) has been extensively studied through approaches such as invariant representation learning (e.g., IRM ( 3 ) , CORAL ( 44 ) ), data augmentation (e.g., MixStyle ( 66 ) , RandConv ( 18 ) ), and learning-based strategies including meta-learning (e.g., MLDG ( 28 ) ), adversarial training (e.g., PAPT ( 55 ) ), and flat-minima regularization (e.g., SWAD ( 9 ) ). Despite their success, these methods primarily operate at the representation level. More recently, CLIP-powered models have advanced DG by providing rich multimodal representations and strong cross-domain transfer. Recent efforts mainly focus on (i) prompt optimization (e.g., CoOp ( 65 ) , CoCoOp ( 64 ) , KgCoOp ( 58 ) , MaPLe ( 25 ) , DPR ( 15 ) , PADG ( 14 ) , SECA ( 23 ) ) and (ii) leveraging CLIP as a backbone for robustness (e.g., StyLIP ( 8 ) , PromptStyler ( 16 ) ). Other works ( 22 ; 13 ; 51 ; 50 ) have also made meaningful progress in this direction. VOLDOGER ( 17 ) focuses on DG data annotation for vision-language tasks, such as image captioning, visual question answering (VQA), and visual entailment (VE). It alleviates the burden of manual annotation by extending LLM-based annotation techniques and provides a reliable benchmark for evaluating DG in these tasks. While effective, these methods remain confined to feature-level invariance and overlook the reasoning processes that underlie generalizable decision-making. Our work addresses this gap by introducing reasoning as a complementary signal for DG. Specifically, we aim to encourage process-level