Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalization
Yuen Chen, Haozhe Si, Guojun Zhang, Han Zhao · 2025
Read Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalization by Yuen Chen, Haozhe Si, Guojun Zhang, Han Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07378.txt.
Introduction
Yuen Chen and co-authors unify ideas for models that must work in new settings in a 2025 paper. A model trained in one hospital or camera set often fails in another. Past fixes aligned gradients or curvature across settings, but they were slow and poorly understood. The authors build a theory that ties them together.
Their moment alignment view shows all three older methods as parts of one rule under a transfer measure. They then design a closed-form algorithm that aligns gradients and curvature without repeated backpropagation. Tests with light and full tuning beat standard training and rival methods. Readers get both insight and a faster tool.
Catalog record
| CREATOR | Yuen Chen, Haozhe Si, Guojun Zhang, Han Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 121 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07378
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalization Yuen Chen, Haozhe Si, Guojun Zhang, Han Zhao Abstract Domain generalization (DG) seeks to develop models that generalize well to unseen target domains, addressing distribution shifts in real-world applications. One line of research in DG focuses on aligning domain-level gradients and Hessians to enhance generalization. However, existing methods are computationally inefficient and the underlying principles of these approaches are not well understood. In this paper, we develop a theory of moment alignment for DG. Grounded in transfer measure , a principled framework for quantifying generalizability between domains, we prove that aligning derivatives across domains improves transfer measure. Moment alignment provides a unifying understanding of Invariant Risk Minimization, gradient matching, and Hessian matching, three previously disconnected approaches. We further establish the duality between feature moments and derivatives of the classifier head. Building upon our theory, we introduce C losed-Form M oment A lignment (CMA), a novel DG algorithm that aligns domain-level gradients and Hessians in closed-form. Our method overcomes the computational inefficiencies of existing gradient and Hessian-based techniques by eliminating the need for repeated backpropagation or sampling-based Hessian estimation. We validate our theory and algorithm through quantitative and qualitative experiments. Table 1: Comparison of our method and prior algorithms. ERM IRM Fish/IGA/AND-Mask Fishr/CORAL HGP/Hutchinson CMA Gradient Matching No Yes Yes No Yes Yes Hessian Matching No No No Yes Yes Yes Closed-Form Hessian – – – No No Yes 1 Introduction Classic machine learning methods rely on the assumption that training and test data are drawn from the same distribution, typically described as being independent and identically distributed ( i.i.d. ). However, the i.i.d. assumption is often violated in real-world scenarios due to variations in sampling populations ( Santurkar et al. 2020 ) , temporal changes ( Shankar et al. 2019 ) , and geographic differences ( Hansen et al. 2013 ; Christie et al. 2018 ) . Performance degradation due to distribution shifts is particularly critical in high-stake applications. For instance, an autonomous driving system ( Dai and Van Gool 2018 ; Hu et al. 2021 ) trained on data collected in the United States may encounter different traffic conditions when deployed in other regions. Similarly, in medical imaging ( Wachinger et al. 2021 ; AlBadawy et al. 2018 ; Tellez et al. 2019 ) , models trained on data from one demographic group may face challenges when applied to a different demographic. Domain generalization (DG) aims to tackle this issue by leveraging data from multiple source domains to learn a model that performs well on unseen but related target domains. Although various approaches have been studied to address the DG problem, including Invariant Risk Minimization (IRM) Arjovsky et al. 2020 , gradient matching ( Shi et al. 2021 ; Koyama and Yamaguchi 2021 ; Parascandolo et al. 2020 ) , Hessian matching ( Rame et al. 2022 ; Hemati et al. 2023 ) , and domain-invariant feature representation learning ( Ben-David et al. 2010 ; Li et al. 2018 ; Tzeng et al. 2017 ; Hoffman et al. 2017 ; Muandet et al. 2013 ; Long et al. 2015 ; Zhao et al. 2019 ) , these methods often appear disconnected and are based on different underlying principles. We discuss these related research in Appendix E . We unify these seemingly disparate methods through the theory of moment alignment. Our theory builds upon transfer measure , a principled DG framework proposed by Zhang et al. 2021 . We first extend the definition of transfer measure to multi-source DG, inducing a target error bound. We then prove that aligning the derivatives improves transfer measure under different assumptions: when there exists a classifier that is simultaneously optimal across all domains (referred to as the IRM assumption ), and when there is not. We show that IRM, gradient matching, and Hessian matching approaches are special cases of moment alignment. Our theory explains the success of state-of-the-art methods like HGP and Hutchinson’s algorithm ( Hemati et al. 2023 ) , which perform both gradient and Hessian matching. This combined approach provides an advantage over methods that only match gradients or Hessians. Furthermore, we establish the duality between feature moments and the derivatives of the classifiers, thereby unifying these approaches. Drawing from the theoretical results, we proposed C losed-Form M oment A lignment (CMA), a novel algorithm to DG that aligns the first- and second-order derivatives across domains. The loss objective in CMA is similar to those of HGP and Hutchinson’s, but CMA enjoys computational efficiency by analytically computing gradients and Hessians. Our method bypasses the computational limitations of existing gradient and Hessian matching techniques that rely on repeated backpropagation or sampling-based estimation. Additionally, we provide two Hessian computation methods—direct Frobenius norm computation for faster performance at higher memory cost, and a memory-efficient method that reduces memory requirement at the expense of increased computation time. This flexibility allows users to balance memory usage and computational time. The empirical evaluation consists of two settings designed to validate our theoretical framework and proposed algorithm. First, we conduct linear probing experiments on Waterbirds, CelebA, and MultiNLI datasets, where the IRM assumption holds. Second, we perform full fine-tuning experiments on selected datasets from the DomainBed benchmark ( Gulrajani and Lopez-Paz 2020 ) , where the IRM assumption may not be satisfied. In the DomainBed experiment, where the IRM assumption is not guaranteed. We compare CMA with ERM, CORAL ( Sun and Saenko 2016 ) , and Fishr ( Rame et al. 2022 ) . CMA’s performance aligns with our theory and matches state-of-the-art performance. Below we summarize our main contributions: • Unified Theory of Moment Alignment: We develop a theory of moment alignment that unifies IRM, gradient matching, and Hessian matching. This unified framework enhances our understanding of the interplay between these methods and their combined effect on improving generalization across domains. We further establish the duality between feature moments and the classifier derivatives. • New Algorithm: We propose C losed-Form M oment A lignment (CMA), a novel DG algorithm that performs both gradient and Hessian matching. CMA enjoys computational efficiency by analytically computing gradients and Hessians, avoiding the need for repeated backpropagation or sampling-based estimation. We offer two Hessian computation methods to optimize memory usage and computational speed. • Empirical Validation: We validate CMA through both quantitative and qualitative analyses. CMA matches state-of-the-art performance while achieving superior worst-group accuracy and feature moment alignment, reducing first- and second-moment discrepancies more effectively than Fishr and ERM. Our work offers a unified perspective that enhances theoretical understanding and practical performance in addressing distribution shifts. As summarized in Table 1 , our method is, to the best of our knowledge, the first to achieve exact gradient and Hessian matching. 2 Preliminaries We consider the problem of DG, where predictors are trained on data drawn from a set of source domains and are evaluated on an unseen target domain. The goal is to learn a predictor that generalizes well to the target domain. Formally, the data are drawn from K source domains \mathcal{S}:=\{\mu_{1},\ldots,\mu_{K}\} and a target domain \mathcal{T}:=\mu_{\mathcal{T}} . Each domain \mu_{i} is a distribution over the input space \mathcal{X} and the label space \mathcal{Y} . The loss of a predictor h:\mathcal{X}\rightarrow\mathcal{Y}