đť•‹ đťź› \mathbb{T^{3}} : Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis

Raza Imam, Hu Wang, Dwarikanath Mahapatra, Mohammad Yaqub · 2025

Read đť•‹ đťź› \mathbb{T^{3}} : Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis by Raza Imam, Hu Wang, Dwarikanath Mahapatra, Mohammad Yaqub (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27265.txt.

Introduction

Raza Imam, Hu Wang, and two other co-authors tackle a trade-off in medical image AI in this 2025 paper. Broad models that pair images with text cope well with new cases but miss fine detail. Expert models tuned to one scan type excel at home yet fail abroad. Merging the two sounds easy, but fixed blends give uneven gains.

Their method, T3, blends the weights of two models at test time, case by case. It sets the blend by how far apart the two models' answers sit, keeping the expert where they agree. A batch version cuts the cost of doing this for each scan. Across four scan types, T3 topped prior methods on accuracy in their tests. The team also sets out a shared test plan for medical merging.

Catalog record

CREATORRaza Imam, Hu Wang, Dwarikanath Mahapatra, Mohammad Yaqub
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-27265

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

𝕋 𝟛 \mathbb{T^{3}} : Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis Raza Imam, Hu Wang, Dwarikanath Mahapatra, Mohammad Yaqub Abstract In medical imaging, vision-language models face a critical duality: pretrained networks offer broad robustness but lack subtle, modality-specific characteristics, while fine-tuned expert models achieve high in-distribution accuracy yet falter under modality shift. Existing model-merging techniques, designed for natural-image benchmarks, are simple and efficient but fail to deliver consistent gains across diverse medical modalities; their static interpolation limits reliability in varied clinical tasks. To address this, we introduce T est- T ime T ask adaptive merging ( 𝕋 𝟛 \mathbb{T^{3}} ), a backpropagation-free framework that computes per-sample interpolation coefficients via the Jensen-Shannon divergence between the two models’ output distributions. 𝕋 𝟛 \mathbb{T^{3}} dynamically preserves local precision when models agree and defers to generalist robustness under drift. To overcome the inference costs of sample-wise merging, we further propose a batch-wise extension, 𝕋 𝟛 ℬ \mathbb{T^{3}}_{\mathcal{B}} that computes merging coefficient across a batch of samples, dramatically reducing computational bottleneck. Recognizing the lack of a standardized medical-merging benchmark, we present a rigorous cross-evaluation protocol spanning in-domain, base-to-novel, and corruptions across four modalities. Empirically, 𝕋 𝟛 \mathbb{T^{3}} sets new state-of-the-art in Top-1 accuracy and error reduction, outperforming strong baselines while maintaining efficiency, paving the way for adaptive MVLM deployment in clinical settings. Our code is available at https://github.com/Razaimam45/TCube . 1 Introduction Medical vision-language models (MVLMs) are typically developed in two flavors: (1) expert models obtained via fine-tuning on domain-specific data, which are highly specialized but may overfit to in-distribution cases, and (2) pretrained models that provide strong generalization but may lack the domain-specific nuances. In healthcare settings, we imagine having two “clinicians” on call: a local specialist, expert MVLM, whose expertise is honed on a single hospital’s scanner, patient demographics, and imaging protocols, and a global generalist, a large-scale pretrained MVLM, whose broad training spans many sites, scanners, and pathologies. The specialist delivers higher confidence in familiar cases but may falter when faced with a case that varies from the norm, e.g., a medical scan from a new device or an unseen patient population. The generalist is typically robust to such distribution shifts but may lack the fine-grained, site-specific nuance, e.g., a specific hospital’s imaging protocols. Clinicians typically take advice from each other, especially on challenging cases, e.g., a neurologist may consult a radiologist on his/her views of a patient’s brain atrophy appearing on MRI scans to be able to diagnose neurodegenerative diseases. This form of multiple views decision making could be mimicked by machine learning models using a simple voting mechanism or more sophisticated model merging methods Wortsman et al. 2022b ; Wortsman et al. 2022a . Existing model-merging strategies typically choose a fixed blend of these two models or rely on simple heuristics that cannot distinguish when the specialist’s local insight truly applies versus when the generalist’s broad knowledge should take precedence. This leaves a critical gap: how to efficiently fuse specialist and generalist decisions to reach to an accurate clinical outcome by utilizing the right aggregation of their knowledge. While prior works in model merging Wortsman et al. 2022b has focused on fixed or globally optimized merging weights, there is limited exploration into dynamic, sample- or batch-wise adaptation strategies for merging expert 1 1 1 We use “domain expert” and “modality expert” interchangeably, where “domain” denotes the data distribution of a specific medical modality. and pretrained models. For instance, Wise-FT Wortsman et al. 2022b merge these models using an average interpolation factor ( i.e: , \alpha =0.5), yet this does not account for the variability in test samples or batches, which may exhibit different levels of domain shift. An integrated solution is required to balance the trade-off between generalization and specialization in VLMs while enhancing zero-shot generalization during inference for a single test sample or a batch. Therefore, weight-interpolation methods Wortsman et al. 2022b ; Lu et al. 2024 ; Oh et al. 2024 ; Yang et al. 2023 remain untested in MVLMs under diagnostic test-time conditions. Medical imaging presents high inter-patient variability, protocol differences, and scarce annotations, calling for a flexible, training-free adaptation rule that balances expert specialization with pretrained robustness. Moreover, without a standardized evaluation protocol, it is difficult to gauge how merging strategies handle domain shifts across medical modalities. In clinical diagnostics, reliable performance across diverse datasets is paramount, far outweighing marginal improvements on any single modality. These gaps motivate our core research questions: Research Questions 1. In what ways can the synergistic performance of pretrained and domain-expert MVLMs be maximized, consistently for various medical tasks? 2. How can pretrained and expert MVLMs balance consensus and divergence in the predicted probability distribution for a given test input? 3. Under conditions of distribution shifts, to what extent do static and dynamic merging methods demonstrate robustness and maintain consistent generalizability across medical modalities? Table 1: Comparison of Static and Dynamic merging methods along three critical dimensions: practicality, domain generalizability, and test-time adaptability. indicates that the method exhibits the desired trait for that criterion, whereas indicates that it does not. \mathbb{T^{3}} excels in all dimensions. Given a pretrained and a expert model, inference cost (I) is measured in forward-passes over the entire test set (with N total samples, grouped into B batches of size BS so that N=B\times\texttt{BS} , where N>>B ). See Section 5.2 for details. Accuracy and Robustness indicates Top-1 Acc and Err (See Section 4 ) averaged over mean OOD of 4 medical datasets. Method Venue Practical (No Training) Universal (Modality-Generalizable) TTA (Adaptive) Consistency (Accuracy \uparrow ) Robustness (Error \downarrow ) Cost ( \# Forwards) Pretrained CLIP - - 38.05 100.0 \mathcal{O}(1B) Expert CLIP - - 55.01 86.7 \mathcal{O}(1B) Model Ensemble - 33.70 111.3 \mathcal{O}(2B) Model Souping PMLR’24 41.82 93.5 \mathcal{O}(1B) Task Arithmetic ICLR’23 44.77 99.4 \mathcal{O}(1B) Slerp NIPS’16 41.82 93.5 \mathcal{O}(1B) Mixup Merging arXiv’25 49.82 84.1 \mathcal{O}(1B) DaWin ICLR’25 44.48 89.0 \mathcal{O}(3B) Sample-wise Merge Ours \mathbb{T^{3}} 58.05 71.9 \mathcal{O} ( N ) Batch-wise Merge Ours \mathbb{T^{3}}_{\mathcal{B}} 58.17 71.7 \mathcal{O}(1B) Our proposed \mathbb{T^{3}} framework is motivated by the need to bridge the gap between specialized and general models when dealing with OOD samples in medical imaging. By learning an adaptive interpolation weight for each sample or batch at test time, \mathbb{T^{3}} dynamically modulates the contributions of the expert and pretrained models. Additionally, the combination of corruption medical datasets (which simulate noise and artifacts) with novel class datasets provides an excellent out-of-distribution benchmark for testing model merging approaches. This comprehensive approach allows us to rigorously evaluate and improve model merging performance in both degraded and unseen conditions. Our contributions can be summarized as follows: – We introduce \mathbb{T^{3}} (pronounced /tee:cube/ ), a non-iterative and backpropagation-free Test-Time Task adaptive interpolation framework that learns optimal batch-wise merging weights without incurring the high computational cost of full backpropagation. – We establish a benchmark for model merging in medical imaging by proposing a hard yet practical cross-evaluation protocol ranging various medical OOD scenarios, assessing across corrupted inputs (MedMNIST-C Di Salvo et al. 2024 ) and novel class generalization (MediMeta Woerner et al. 2024b ). – We provide an empirical analysis that justifies the test-time dynamic model merging and its benefits in addressing distribution shifts consistently across multiple medical tasks. – We

Read the complete text