GeneralizeFormer: Layer-Adaptive Model Generation across Test-Time Distribution Shifts

Sameer Ambekar, Zehao Xiao, Xiantong Zhen, Cees G. M. Snoek · 2025

Read GeneralizeFormer: Layer-Adaptive Model Generation across Test-Time Distribution Shifts by Sameer Ambekar, Zehao Xiao, Xiantong Zhen, Cees G. M. Snoek (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12195.txt.

Introduction

Sameer Ambekar and co-authors build a small transformer that writes fresh model parameters on the fly, so a vision model can meet new domains at test time without retraining. Their 2025 paper works in domain generalization: train on several sources, then face targets never seen. Standard fixes fine-tune or nudge the classifier online, which costs time and can forget the source.

Their generator runs per target batch during inference. It rewrites only the normalization layers and the final classifier, leaving the heavy convolutional weights fixed, which keeps the cost low. Layer-wise gradients steer the adaptation to each shift. Six benchmark sets showed the method handling varied shifts, staying nimble across changing targets, and holding onto source knowledge.

Catalog record

CREATORSameer Ambekar, Zehao Xiao, Xiantong Zhen, Cees G. M. Snoek
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12195

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GeneralizeFormer: Layer-Adaptive Model Generation across Test-Time Distribution Shifts Sameer Ambekar, Zehao Xiao, Xiantong Zhen, Cees G. M. Snoek Abstract We consider the problem of test-time domain generalization, where a model is trained on several source domains and adjusted on target domains never seen during training. Different from the common methods that fine-tune the model or adjust the classifier parameters online, we propose to generate multiple layer parameters on the fly during inference by a lightweight meta-learned transformer, which we call GeneralizeFormer . The layer-wise parameters are generated per target batch without fine-tuning or online adjustment. By doing so, our method is more effective in dynamic scenarios with multiple target distributions and also avoids forgetting valuable source distribution characteristics. Moreover, by considering layer-wise gradients, the proposed method adapts itself to various distribution shifts. To reduce the computational and time cost, we fix the convolutional parameters while only generating parameters of the Batch Normalization layers and the linear classifier. Experiments on six widely used domain generalization datasets demonstrate the benefits and abilities of the proposed method to efficiently handle various distribution shifts, generalize in dynamic scenarios, and avoid forgetting. Our code is available: https://github.com/ambekarsameer96/generalizeformer 1 Introduction To counter the performance degradation of deep learning algorithms when test data strays from the training data distribution, the framework of test-time adaptation emerged [ 60 , 29 , 77 ] . In test-time adaptation [ 66 ] , one trains a model only on source domains and adapts the source-trained model during inference on (unseen) target data. Various methods are proposed for adaptation to a single source distribution, as typical for image corruption [ 56 , 60 , 39 ] , and for multiple source distributions, as common in domain generalization [ 29 , 68 , 77 ] . In this paper, we focus on the latter scenario, referring to it as test-time domain generalization. Figure 1 : Illustration of test-time generalization methods. (a) Fine-tuning methods update the model online with large batches of samples. (b) Classifier adjustment methods feedforwardly update the classifier according to the test data. (c) Our method adaptively generates target-specific parameters in different layers according to the target distribution, enabling it to handle various distribution shifts without fine-tuning. Current test-time generalization methods focus on fine-tuning the model with unsupervised loss functions on the target data [ 60 , 21 ] or adjusting the linear classifiers during inference [ 29 , 77 ] . However, these methods utilize batches of target data at the same time [ 38 , 32 ] or in an online manner [ 60 ] . The alternative of classifier adjustment only considers the distribution shifts during classification, but ignores shifts resulting from feature extraction [ 79 , 33 ] . Moreover, these methods rely on online updates of model parameters on each specific target distribution and tend to forget the discriminative ability of previous source distributions, which makes them struggle in complex or dynamic scenarios [ 73 , 77 ] . Recently, Lee et al . [ 33 ] propose to handle different distribution shifts by manually selecting and fine-tuning layers, which, however, still requires different selection and fine-tuning operations for different specific target distributions. Building on these problems and observations, we propose to adaptively generate target-specific parameters in different model layers to handle various distribution shifts without re-training the parameters (see also Fig. 1 ). We propose GeneralizeFormer , a transformer that generates target-specific layer-wise model parameters for each target batch at test-time. The target parameters are generated from the source-trained parameters, the target features, and the layer-wise gradient information. With the layer-wise generation of the target parameters, the method can adaptively handle distribution shifts in different feature levels without manually selecting and fine-tuning parameters. To enable model generation ability across distribution shifts, we train the transformer under a meta-learning strategy. At test time, GeneralizeFormer achieves adaptation without online optimization of the model, making it especially effective in dynamic scenarios that suffer from multiple target distributions. This also helps the model to preserve the source model information and avoid forgetting after model adaptation . To reduce the computational and time costs for adaptation , we fix the parameters of the convolutional layers while generating the parameters of all Batch Normalization layers and that of the classifiers in the network. Experiments on six domain generalization benchmarks and various distribution shifts show the ability of our method to handle source forgetting, dynamic scenarios, and multiple distribution shifts. 2 Related Work Domain adaptation and domain generalization. Distribution shifts have been widely investigated in domain adaptation [ 42 , 64 , 71 ] and domain generalization methods [ 3 , 82 , 62 ] . Domain adaptation deals with the distribution shifts between training and test data by assuming the accessible labeled or unlabeled target samples during training [ 6 , 16 , 9 , 23 ] . However, in real applications, the target data is usually unknown during training [ 78 ] . To solve this problem, domain generalization is proposed to train the model only on source domains and evaluate it on unseen target domains [ 45 , 36 , 10 , 22 , 65 , 48 ] . The problem of domain generalization is the lack of target information of the source-trained model, which may lead to the adaptivity gap [ 13 ] . Recently, source-free domain adaptation has emerged [ 38 , 32 , 61 , 51 , 15 , 31 , 72 , 58 ] where source model is adapted to target domains at test time. Our test-time domain generalization setting is related to the above settings, where we train the model on several source domains as domain generalization. Rather than adapting the model to the entire target domain, we achieve adaptation at each test step. Test-time adaptation and generalization. Test-time adaptation is also an emerging method for handling distribution shifts at test time [ 56 , 60 , 40 , 18 , 47 , 57 , 41 , 69 , 74 ] . These methods usually train the model on a single source distribution and achieve adaptation at test time in an online manner, focusing on covariate shifts such as corruptions [ 60 , 21 , 75 , 46 ] . The idea has also been applied to the domain generalization setting, where multiple source domains are available during training [ 12 , 34 , 67 , 30 , 77 , 2 , 1 ] . We refer to this setting as test-time domain generalization. One commonly used test-time method is fine-tuning the source-trained model with an unsupervised loss like entropy minimization [ 60 , 75 , 81 , 21 , 74 , 46 ] . As an alternative, changing the statistics of the Batch Normalization layers is also widely utilized [ 53 , 12 , 70 , 25 , 20 , 39 ] . Another method for test-time adaptation and generalization is classifier adjustment [ 29 , 68 , 77 ] , where the prototypes or the last fully-connected layer of the model is adjusted at test time, without fine-tuning. Our method focuses on the test-time domain generalization setting. Lee et al . [ 33 ] showed that fine-tuning different layers benefits different

Read the complete text