Balanced Direction from Multifarious Choices: Arithmetic Meta-Learning for Domain Generalization

Xiran Wang, Jian Zhang, Lei Qi, Yinghuan Shi · 2025

Read Balanced Direction from Multifarious Choices: Arithmetic Meta-Learning for Domain Generalization by Xiran Wang, Jian Zhang, Lei Qi, Yinghuan Shi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18987.txt.

Introduction

Xiran Wang, Jian Zhang, Lei Qi, and Yinghuan Shi wrote this 2025 paper on domain generalization, the task of learning that still works on new kinds of data. Models often fail when test data come from a different mix than the training data. Meta-learning methods fight this by matching gradients across source domains, which keeps the model from leaning too hard on any one domain.

The authors show that gradient matching can be reached by many paths, and current methods take just one. They also show a second need: the balanced settings should sit near the center of the best settings for each domain. Their fix is simple. They weight the gradients by arithmetic rules while keeping the matching idea in place. The method aims at settings that are both balanced across domains and close to each domain's own best point.

Catalog record

CREATORXiran Wang, Jian Zhang, Lei Qi, Yinghuan Shi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18987

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Balanced Direction from Multifarious Choices: Arithmetic Meta-Learning for Domain Generalization Xiran Wang, Jian Zhang, Lei Qi, Yinghuan Shi Abstract Domain generalization is proposed to address distribution shift, arising from statistical disparities between training source and unseen target domains. The widely used first-order meta-learning algorithms demonstrate strong performance for domain generalization by leveraging the gradient matching theory, which aims to establish balanced parameters across source domains to reduce overfitting to any particular domain. However, our analysis reveals that there are actually numerous directions to achieve gradient matching, with current methods representing just one possible path. These methods actually overlook another critical factor that the balanced parameters should be close to the centroid of optimal parameters of each source domain. To address this, we propose a simple yet effective arithmetic meta-learning with arithmetic-weighted gradients. This approach, while adhering to the principles of gradient matching, promotes a more precise balance by estimating the centroid between domain-specific optimal parameters. Experimental results validate the effectiveness of our strategy. Our code is available at https://github.com/zzwdx/ARITH . 1 Introduction Figure 1 : In the ternary parameter space (x,y,z) , gradient matching can be deduced in any directions of the yellow surface. The black arrow is the updating direction of existing methods, which moves away from the optimal solution of domain 1. Deep neural networks Krizhevsky et al. 2017 usually hinge on the premise that both training and test data are independent and identically distributed (i.i.d.). However, this assumption often fails in dynamic real-world contexts, resulting in performance degradation when test data diverges from the distribution encountered during training Li et al. 2019 . This shift in distribution proves detrimental, as the model tends to overfit specific representations that may be absent or inadequately represented during the test phase Shah et al. 2020 ; Izmailov et al. 2022 ; Qiu et al. 2024 . In recent years, the domain generalization (DG) Wang et al. 2022 paradigm has been widely studied to deal with distribution shift, which refers to leveraging multiple source domains to develop a model with the generalization ability that can be directly applied to arbitrary unseen target domains. Some studies identify invariance as a critical factor for domain generalization, which assume that certain stable elements persist across source domains and can also apply to unseen target domains. These methods often employ techniques such as data augmentation Zhou et al. 2020 ; Li et al. 2021 ; Guo et al. 2023 and adversarial learning Li et al. 2018b ; Dayal et al. 2024 to capture invariant features, while others impose more intricate forms of invariance at gradient Shahtalebi et al. 2021 ; Wang et al. 2023a or predictor Arjovsky et al. 2020 ; Krueger et al. 2021 level. However, the presumed invariance doesn’t always translate to improved generalization due to the random nature of target domains. As illustrated in Gulrajani and Lopez-Paz 2020 , only a few of them surpass vanilla empirical risk minimization (ERM) Naumovich and Vlamimir 1998 in terms of average accuracy across multiple standard datasets. Rather than strictly enforcing an invariance factor, balance -based methods Dou et al. 2019 ; Cha et al. 2021 ; Rame et al. 2022b take a more flexible way by ensuring that model parameters are balanced across various domains. For example, meta-learning Li et al. 2018a ; Zhang et al. 2020 ; Shu et al. 2021 aims to achieve an optimal balance among source domains by implicitly guiding their gradient directions called gradient matching . The rationale is that large angles between gradients indicate conflicting objectives, suggesting that updating one domain may negatively impact the optimization process of others. In contrast, smaller gradient angles imply that optimizing one domain does not disrupt other domains, allowing for a mutually beneficial outcome by optimizing their combined gradient, thereby mitigating the risk of overfitting to specific domains Shi et al. 2021 ; Wang et al. 2023b . Although gradient matching currently serves as the basic theory in meta-learning for domain generalization, we believe it has not been fully explored. As shown in Fig. 1 , each red arrow represents a gradient updating process. While existing methods often select the direction of g_{1}+g_{2} , there are numerous other ways in the yellow region that can also achieve gradient matching. Alternatively, the centroid of a structure is known for its balanced nature. In Fig. 3 (d)(h), updating direction towards the centroid of source domain experts, marked by the yellow color, tend to lie closer to the target domain’s loss basin Izmailov et al. 2018 ; Cha et al. 2021 with better generalization ability than those in the g_{1}+g_{2} direction. This suggests that focusing only on gradient matching without considering balanced positioning seems insufficient. We propose an arithmetic meta-learning framework by adjusting the weights of g_{1} and g_{2} in Fig. 1 to identify a more balanced position between source domains, where the weights of gradients are selected to form an arithmetic progression. Intuitively, since g_{2} is less correlated with the initial model than g_{1} , it is reasonable to assign g_{2} with smaller weights. We further demonstrate that a set of arithmetically decreasing gradient weights not only follows the principle of gradient matching, but also reflects model averaging Izmailov et al. 2018 that indirectly estimate the centroid of domain experts. This centroid approximation is expected to achieve a more accurate balance across source domains to enhance model’s generalization stability in the unseen target scenarios. Our contribution can be summarized as follows: • We prove that existing first-order meta-learning strategies for domain generalization represent just one of many possible directions for gradient matching, and they overlook the need to balance the model’s position from the optimal parameters of each source domain. • We integrate model averaging into meta-learning and propose an arithmetic gradient-based strategy to simulate this process, which aims to estimate the centroid of domain-specific optimal parameters while ensuring consistency in gradient direction. Our method is simple to implement, requiring adjustments of one line in Algorithm 1 . • Arithmetic meta-learning outperforms traditional meta-learning strategies across multiple domain generalization benchmarks, and shows synergistic potential when integrated with global averaging techniques. Algorithm 1 Arithmetic Meta-Learning for DG 0: Domains \{\mathcal{D}_{1},\mathcal{D}_{2},...,\mathcal{D}_{S}\} ; model parametrized by \Theta ; hyperparameters k and \epsilon 1: for iterations =1,2,... do 2: Initialize parameters \theta_{1}\leftarrow\Theta ; 3: for \mathcal{D}_{i}\in\{\mathcal{D}_{1},\mathcal{D}_{2},...,\mathcal{D}_{S}\} do 4: Perform k gradient updates from \mathcal{D}_{i} to obtain \theta_{i+1}\leftarrow\theta_{i} with g_{i}=\theta_{i}-\theta_{i+1} 5: end for 6: \Theta\leftarrow\Theta-\epsilon\sum_{i=1}^{n}g_{i} // for Fish 7: \Theta\leftarrow\Theta-\frac{1}{n+\epsilon}\sum_{i=1}^{n}(n+1-i)g_{i} // for Arith 8: end for 2 Method 2.1 Preliminary Problem setting. In domain generalization, we are provided with source domains \mathcal{S}=\{\mathcal{D}_{1},\mathcal{D}_{2},...,\mathcal{D}_{S}\} and unseen target domains \mathcal{T}=\{\mathcal{D}_{S+1},\mathcal{D}_{S+2},...,\mathcal{D}_{S+T}\} . The s -th domain consisting of N_{s} samples is represented as \mathcal{D}_{s}=\{(x^{s}_{i},y^{s}_{i})\}_{i=1}^{N_{s}} , where x^{s}_{i} denotes the i -th sample and y^{s}_{i} refers to its corresponding label. Our goal is to leverage these source domains \mathcal{S} to develop a model capable of seamlessly generalizing to any unseen target domain \mathcal{T} . First-order meta-learning Nichol et al. 2018 segments the optimization process into an inner loop and an outer loop. Each iteration can be summarized as follows: a task comprises a batch of data sampled from a specific data distribution, and a step aggregates multiple tasks to form a larger batch. In domain

Read the complete text