Why Domain Generalization Fail? A View of Necessity and Sufficiency

Tung-Long vuong, Vy Vo, Hien Dang, Van-Anh Nguyen, Thanh-Toan Do, Mehrtash Harandi, Trung Le, Dinh Phung · 2025

Read Why Domain Generalization Fail? A View of Necessity and Sufficiency by Tung-Long vuong, Vy Vo, Hien Dang, Van-Anh Nguyen, Thanh-Toan Do, Mehrtash Harandi, Trung Le, Dinh Phung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10716.txt.

Introduction

Long-Tung Vuong and co-authors ask why models that should work on new data so often fail, in a 2025 study. The field is domain generalization: a model trained on some data worlds must hold up in a fresh one. In tests, fancy methods often fail to beat a plain base method.

The team sorts the needs for success into two kinds: must-have conditions and nice-to-have ones. Past methods act as brakes that chase the nice kind, while they may break the must kind and lose the prize. With few training worlds, the nice kind cannot even be checked. The team then gives a fix that guards the must kind while it lifts the nice one. Tests on stock sets back the claim.

Catalog record

CREATORTung-Long vuong, Vy Vo, Hien Dang, Van-Anh Nguyen, Thanh-Toan Do, Mehrtash Harandi, Trung Le, Dinh Phung
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10716

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why Domain Generalization Fail? A View of Necessity and Sufficiency Tung-Long vuong, Vy Vo, Hien Dang, Van-Anh Nguyen, Thanh-Toan Do, Mehrtash Harandi, Trung Le, Dinh Phung Abstract Despite a strong theoretical foundation, empirical experiments reveal that existing domain generalization (DG) algorithms often fail to consistently outperform the ERM baseline. We argue that this issue arises because most DG studies focus on establishing theoretical guarantees for generalization under unrealistic assumptions, such as the availability of sufficient, diverse (or even infinite) domains or access to target domain knowledge. As a result, the extent to which domain generalization is achievable in scenarios with limited domains remains largely unexplored. This paper seeks to address this gap by examining generalization through the lens of the conditions necessary for its existence and learnability. Specifically, we systematically establish a set of necessary and sufficient conditions for generalization. Our analysis highlights that existing DG methods primarily act as regularization mechanisms focused on satisfying sufficient conditions, while often neglecting necessary ones. However, sufficient conditions cannot be verified in settings with limited training domains. In such cases, regularization targeting sufficient conditions aims to maximize the likelihood of generalization, whereas regularization targeting necessary conditions ensures its existence. Using this analysis, we reveal the shortcomings of existing DG algorithms by showing that, while they promote sufficient conditions, they inadvertently violate necessary conditions. To validate our theoretical insights, we propose a practical method that promotes the sufficient condition while maintaining the necessary conditions through a novel subspace representation alignment strategy. This approach highlights the advantages of preserving the necessary conditions on well-established DG benchmarks. 1 Introduction Domain generalization (DG) aims to train a machine learning model on multiple data distributions so that it can generalize to unseen data distributions. Although challenging, DG is crucial for practical scenarios where there is a need to quickly deploy a prediction model on a new target domain without access to target data. Various approaches have been proposed to address the DG problem, which can be broadly categorized into 4 families: representation alignment, invariant prediction, augmentation and ensemble learning. Representation alignment which is established based on domain adaptation ( Ben-David et al., 2010 ; Ben-Hur et al., 2001 ; Phung et al., 2021 ; Zhou et al., 2020 ; Johansson et al., 2019 ) , mainly discuss the differences between source and target domains and focuses on learning domain-invariant representations by reducing the divergence between latent marginal distributions ( Long et al., 2017 ; Ganin et al., 2016 ; Nguyen et al., 2021 ; Shen et al., 2018 ; Xie et al., 2017 ; Ilse et al., 2020 ) or class-conditional distributions ( Gong et al., 2016 ; Li et al., 2018b ; Tachet des Combes et al., 2020 ) . Invariant prediction approaches, grounded in causality theory, ensure stable performance regardless of the domain by learning a consistently optimal classifier ( Arjovsky et al., 2020 ; Ahuja et al., 2020 ; Krueger et al., 2021 ; Li et al., 2022a ; Mitrovic et al., 2020 ; Zhang et al., 2023 ) . Data augmentation applies predefined or learnable transformations on the original samples or their features to create augmented data, thereby enhancing the model’s generalization capabilities ( Mitrovic et al., 2020 ; Wang et al., 2022b ; Zhou et al., 2020 ; Zhou et al., 2021 ; Zhang et al., 2017 ; Wang et al., 2020b ; Zhao et al., 2020 ; Yao et al., 2022a ; Carlucci et al., 2019 ; Yao et al., 2022b ) . With sufficient and diverse causal-preserving transformations, the model’s generalization can be effectively guaranteed ( Wang et al., 2020b ) . Recently, Ensemble Learning , which explicitly involves training multiple instances of the same architecture with different initializations or splits of the training data then combining their predictions ( Zhou et al., 2021 ; Ding & Fu, 2017 ; Zhou et al., 2021 ; Wang et al., 2020a ; Mancini et al., 2018 ; Cha et al., 2021 ; Arpit et al., 2022 ) . Alternatively, implicit ensemble methods that approximate ensembling by averaging model weights (WA) from training trajectories (e.g., checkpoints at different time steps) have been shown to significantly enhance robustness under domain shifts ( Izmailov et al., 2018 ; Cha et al., 2021 ; Rame et al., 2022 ; Wortsman et al., 2022 ) . Despite their strong performance in DG, most theoretical analyses of ensemble learning remain concentrated on the perspective of flatness of the loss landscape, or uncertainty-aware frameworks, leaving the connection between ensemble and DG largely underexplored. Although the conventional DG algorithms ( representation alignment, invariant prediction, data augmentation ) are developed with strong theoretical foundations, these methods have not consistently outperformed Empirical Risk Minimization (ERM) on fair model selection criteria ( Gulrajani & Lopez-Paz, 2021 ; Idrissi et al., 2022 ; Ye et al., 2022 ; Chen et al., 2022b ) . In contrast, Ensemble -based approaches (e.g., SWAD ( Cha et al., 2021 ) ) demonstrate a substantial performance improvement over the ERM and other conventional DG algorithms by a large margin. We argue that this phenomenon arises because the theoretical frameworks behind conventional DG algorithms typically establish generalization under conditions where target domains are either known or sufficiently diverse, or when a large number of training domains are available. As a result, the extent to which domain generalization can be achieved in practical scenarios, characterized by a limited and finite number of domains, remains elusive unexplored. Table 1: Summary of Conditions for Generalization Condition Type Target DG approach Label-identifiability ( 2.1 ) Assumption Causal support ( 2.3 ) Assumption Optimal hypothesis for training domains ( 3.1 ) Necessary Optimal hypothesis for training domains + Invariant representation function ( 3.2 .1) Sufficient Representation alignment Optimal hypothesis for training domains + Sufficient and diverse domains ( 3.2 .2) Sufficient Invariant prediction Optimal hypothesis for training domains + Invariance-preserving transformations ( 3.2 .3) Sufficient Data augmentation Invariance-preserving representation function ( 3.3 ) Necessary Ensembles Our work aims to fill in this gap with a comprehensive study of DG in scenarios with limited training domains through the lens of necessary and sufficient conditions for achieving generalization. The contributions of this paper are summarized as follows: 1. DG through the lens of Necessity and Sufficiency. We systematically establish a set of necessary and sufficient conditions for generalization, highlighting that existing DG methods act as regularization mechanisms that predominantly focus on satisfying sufficient conditions while often neglecting necessary ones (see Section 3 .1). However, sufficient conditions are non-verifiable with limited training domains. In such cases, we argue that regularization targeting sufficient conditions aims to maximize the likelihood of generalization, while regularization targeting necessary conditions ensures its existence (see Section 3 .2). 2. Why do conventional DG algorithms fail? In Section 4 , we explain why conventional DG algorithms fail to consistently outperform the ERM baseline by demonstrating that, while they promote sufficient conditions, they may inadvertently violate necessary conditions. We also establish a connection between the necessary conditions and the recent ensemble strategy, demonstrating that ensemble methods indeed encourage models to satisfy these conditions. 3. Preserving Necessity in Representation Alignment. Finally, we empirically validate our theories by introducing a practical method that encourages the sufficient condition without

Read the complete text