Test-Time Warmup for Multimodal Large Language Models
\Name Nikita Rajaneesh \Email [email protected] \addr Columbia University \Name Thomas Zollo \Email [email protected] \addr Columbia University \Name Richard Zemel \Email [email protected] \addr Columbia University · 2025
Read Test-Time Warmup for Multimodal Large Language Models by \Name Nikita Rajaneesh \Email [email protected] \addr Columbia University \Name Thomas Zollo \Email [email protected] \addr Columbia University \Name Richard Zemel \Email [email protected] \addr Columbia University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10641.txt.
Introduction
Nikita Rajaneesh and 2 co-authors propose warming up a multimodal model at test time in this 2025 paper. A multimodal large language model joins a vision encoder, a connector, and a language model, each pre-trained on huge data. The joined system, though, is usually trained on far fewer examples, which leaves it weak on hard reasoning that mixes text and images.
Their Test-Time Warmup adapts the model separately for each test item, using weakly supervised side tasks tied to that item rather than new labeled data. On the Llama vision model they tested, scores rose on three benchmarks covering general multimodal reasoning, medical visual questions, and visual question answering. The gains show per-item preparation can add robustness.
Catalog record
| CREATOR | \Name Nikita Rajaneesh \Email [email protected] \addr Columbia University \Name Thomas Zollo \Email [email protected] \addr Columbia University \Name Richard Zemel \Email [email protected] \addr Columbia University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-10641
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Test-Time Warmup for Multimodal Large Language Models \Name Nikita Rajaneesh \Email [email protected] \addr Columbia University \Name Thomas Zollo \Email [email protected] \addr Columbia University \Name Richard Zemel \Email [email protected] \addr Columbia University Abstract Multimodal Large Language Models (MLLMs) hold great promise for advanced reasoning at the intersection of text and images, yet they have not fully realized this potential. MLLMs typically integrate an LLM, a vision encoder, and a connector that maps the vision encoder’s embeddings into the LLM’s text embedding space. Although each component is pretrained on massive datasets with billions of samples, the entire multimodal model is typically trained on only thousands (or a few million) samples, which can result in weak performance on complex reasoning tasks. To address these shortcomings, instead of relying on extensive labeled datasets for fine-tuning, we propose a Test-Time Warmup method that adapts the MLLM per test instance by leveraging data from weakly supervised auxiliary tasks. With our approach, we observe a relative performance improvement of 4.03% on MMMU, 5.28% on VQA-Rad, and 1.63% on GQA on the Llama-Vision-Instruct model. Our method demonstrates that ‘warming up’ before inference can enhance MLLMs’ robustness across diverse reasoning tasks. 1 Introduction In recent years, large language models (LLMs) have taken center stage in natural language processing, achieving remarkable results across a wide range of tasks ( Minaee et al. 2025 ; Brown et al. 2020 ; Rae et al. 2022 ; Touvron et al. 2023 ) . Concurrently, vision foundation models such as CLIP have redefined image understanding ( Radford et al. 2021 ) . Building on these advances, multimodal large language models (MLLMs) ( ( Liu et al. 2023 ; Grattafiori et al. 2024 ) ) have shown promising performance in multimodal tasks such as image captioning ( Li et al. 2023 ) and visual question answering ( McKinzie et al. 2024 ) , offering a vision of a future where advanced reasoning is robust across text and image. MLLMs typically integrate an LLM, a vision encoder (e.g., CLIP), and a connector that maps the vision encoder’s embeddings into the LLM text embedding space. Despite the power of their individual components, each trained on billions of samples, for the final instruction tuning step, MLLMs are often limited by a relatively small amount of multimodal training data, typically ranging from only a few thousand to a few million examples. This lack of large-scale, end-to-end multimodal supervision can hinder their ability to tackle complex reasoning tasks. Recent work has shown that MLLMs are vulnerable to distribution shifts, which undermine their reasoning capabilities ( Wang et al. 2024 ) , and they can even exhibit catastrophic forgetting and hallucinations ( Zhai et al. 2023 ) . Furthermore, research indicates that MLLMs struggle with specialized tasks, such as interpreting diagrams, due to limited task-specific training data ( Joshi et al. 2025 ) . Although fine-tuning is an effective method for addressing these challenges, the scarcity or difficulty of obtaining labeled data often makes supervised adaptation impractical in many real-world scenarios, such as medical applications. To avoid relying on extensive labeled datasets, we introduce a novel Test-Time Warmup (TTW) method which adapts the MLLM on the fly with minimal supervision. Our method, per test instance, leverages data from weakly supervised auxiliary tasks to guide adaptation, allowing the model to refine its representations for complex reasoning without requiring any ground truth annotations. More specifically, we adapt the MLLM individually to each test instance by performing gradient updates on a set of auxiliary tasks derived from that instance. After adaptation, we perform inference on the relevant task instance and discard the updated model weights. Figure 1: An example demonstrating Test-Time Warmup improving a model’s attention to detail (a crucial aspect of perceptual reasoning). In step (1), per auxiliary task prompt listed in Figure 2, we generate 10 caption-like outputs. In this example, we show the outputs generated for the object detection auxiliary task. Then, we use the maximum CLIP score to choose one caption. Here, the caption with more details is chosen because it’s more aligned with the image. Then in step (2), we perform gradient steps on the chosen caption-like outputs for N auxiliary task prompts ( N=10 as shown in Figure 2). In doing so, we enforce the model to pay attention to all the objects, including the mask, in the image. For inference, in step (3), the “warmed-up” MLLM is better informed to answer the question because it has paid attention to the mask. In this work, we focus on evaluating Test-Time Warmup and multimodal large language models (MLLMs) on visual question answering tasks using datasets such as MMMU ( Yue et al. 2024 ) , GQA ( Hudson and Manning 2019 ) and VQA-Rad ( Narayanan et al. 2024 ) . Through our evaluation of a popular open-source MLLM, we identify a set of failure modes indicating that these models often struggle when provided with tasks that (1) deviate substantially from the bulk of their training data, or (2) require a highly detailed understanding of the visual content. To address these weaknesses, we propose a set of auxiliary tasks to refine image representations and “warmup” the model for inference. The auxiliary tasks are designed as input prompts to the target model and are independent of the downstream target tasks. An illustrative analogy is a “nurse practitioner and doctor” scenario: the nurse practitioner begins by asking the patient a standardized set of questions, unrelated to any specific ailment, and records the responses for the doctor. With a richer baseline of information about the patient, the doctor can then make a more accurate diagnosis. Figure 1 provides an example of how Test-Time Warmup can prevent an MLLM from making errors. In our experiments, we find that our method outperforms baselines on datasets that focus on perceptual reasoning rather than domain knowledge. On MMMU, our approach shows gains for plot, chart or tables images: a form of data that constitutes only a small fraction of the typical training corpus and demands a deeper level of visual comprehension. In the case of GQA, which involves image-question pairs necessitating compositional reasoning, our method boosts performance for scenes that contain many subtle details. Similarly, on VQA-Rad, we see performance improvement on questions that require advanced image understanding. We attribute these gains to our method’s ability to foster more nuanced image representations and bring forth existing knowledge in the model rather than provide additional information to the model. Our main contributions include the following: • A Test-Time Warmup method for multimodal large language models that uses data from a set of auxiliary tasks to perform gradient steps on the MLLM per test input. After adaptation, we perform inference on the test instance and discard the updated model weights. • Using our method, we show a relative improvement of 4.03% on MMMU, 5.28% on VQA-Rad and 1.63% on GQA on Llama-Vision-Instruct model ( Grattafiori et al. 2024 ) . Although we only show results for visual question answering tasks, through this work we motivate the use of our method to perform a diverse set of reasoning tasks. In the case of web agents, for example, performing gradient steps on a fixed set of questions and answers based