Learning Subset-Shared Invariances for Domain Generalization with Mixture-of-Experts
Tien-Hung Nguyen, Tien-Dat Tran, M.-Duong Nguyen, Kok-Seng Wong · 2026
Read Learning Subset-Shared Invariances for Domain Generalization with Mixture-of-Experts by Tien-Hung Nguyen, Tien-Dat Tran, M.-Duong Nguyen, Kok-Seng Wong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25665.txt.
Introduction
A model trained in several hospitals or climates should work in a new one it has never seen, the promise of domain generalisation. The standard trick forces the model's inner representation to look identical across all training domains. Tien-Hung Nguyen, Tien-Dat Tran, M.-Duong Nguyen, and Kok-Seng Wong argue in 2026 that this demand backfires. Each added domain shrinks the space of allowed representations. Useful predictive clues, stable in some domains but not all, get thrown away. Their alternative assumes structure is shared only within subsets of domains.
A mixture of experts implements the idea: each expert aligns just the domains it serves, and a router assembles the subset invariant parts for each prediction. Training goals push for selective alignment, confident balanced routing, and experts that genuinely differ. On DomainBed benchmarks the method generalised better out of domain and held up as the domains grew more varied.
Catalog record
| CREATOR | Tien-Hung Nguyen, Tien-Dat Tran, M.-Duong Nguyen, Kok-Seng Wong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 102 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25665
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Subset-Shared Invariances for Domain Generalization with Mixture-of-Experts Tien-Hung Nguyen, Tien-Dat Tran, M.-Duong Nguyen, Kok-Seng Wong Abstract Domain generalization (DG) aims to learn a model from one or more source domains that generalizes to an unseen target domain without accessing target data during training. A common approach enforces invariance of representations across all source domains, assuming predictive structure is globally shared. However, we demonstrate that enforcing invariance across more domains gradually restricts the feasible representation space, discarding transferable predictive factors that are not universally shared. To address this limitation, we propose subset-shared invariance, where predictive structure is assumed stable only within domain subsets. We implement this principle with a mixture-of-experts architecture, where each expert aligns the specific domains it serves and a routing mechanism composes subset-invariant components for prediction. This creates a routing-conditioned invariance, jointly learned with the representation. To facilitate effective decomposition, we develop training objectives that encourage selective alignment, confident and balanced routing, and diverse expert specialization. Experiments on DomainBed benchmarks demonstrate improved out-of-domain generalization and greater robustness under increasing domain heterogeneity. Our results suggest that DG should move beyond enforcing a single global invariance and instead model invariance through partially shared structure across domain subsets. 1 Introduction Domain generalization (DG) refers to the challenge of training models that perform well on unseen domains, without having access to data from these target domains during training [ 46 , 23 , 34 ] . A widely used approach in DG is to develop domain-invariant representations by ensuring that the conditional distribution P(Z|Y) remains consistent across different domains. This idea is central to many methods, including feature alignment [ 57 , 51 ] and invariant learning frameworks such as invariant risk minimization (IRM) [ 1 ] and its extensions [ 53 , 56 ] . Nonetheless, applying invariance uniformly to all domains may be too restrictive (see Figure 1 ). In real-world settings, domains may only partially overlap in structure. For instance, specific features could be stable in some domains but vary in others [ 52 , 68 ] . Therefore, imposing global invariance can eliminate useful predictive information, especially when domain shifts are heterogeneous [ 21 , 62 ] . This observation indicates that predictive structure is not universally shared, but instead distributed among subsets of domains. Similar patterns of subset-based heterogeneity are also found in multimodal learning, where individual samples may use different modality combinations [ 71 ] . To formalize this limitation, we analyze the effect of enforcing invariance across an increasing number of domains and show that it progressively reduces the mutual information between learned representations and labels. This reveals an inherent trade-off between invariance and predictive information. As invariance constraints become stronger, the representation becomes less informative for prediction. This observation suggests that DG should not enforce invariance globally, but should instead selectively preserve predictive structure that is stable only within subsets of domains. We therefore argue that invariance in DG should be structured rather than global . Specifically, we introduce subset-shared invariance , where invariant structure is assumed to hold within subsets of domains rather than across all domains. This reframes DG as a structured decomposition problem. Instead of learning a single invariant representation, the model should capture multiple invariant components, each corresponding to a subset of domains, and combine them adaptively for prediction. To instantiate this principle, we leverage a mixture-of-experts (MoE) architecture [ 30 , 55 ] as a mechanism for learning such a decomposition. A routing function induces a soft partition over inputs, which in turn defines subsets of domains associated with each expert. Figure 1 : Test accuracy changes as the number of source domains increases under a fixed training budget. Adding more domains at first improves performance, but after a certain point, accuracy begins to decline. This pattern shows an important limitation of enforcing global invariance. When more heterogeneous domains are included, the shared invariant signal weakens and valuable predictive information is lost. In contrast, MESSI stays robust, implying that preserving subset-specific structure is crucial for generalization when domains differ significantly. We enforce invariance only within the subsets defined by routing, allowing the model to learn in which domains invariance is appropriate. This results in a routing-induced invariance , where the model learns where invariance should hold, rather than enforcing it uniformly across all domains. MoE models have been used in DG to boost both capacity and specialization [ 35 , 9 , 14 ] . However, earlier work relies on experts to act as feature extractors or domain-specific modules. Our method instead uses routing to directly shape how invariance is applied, so that expert specialization helps find invariant relationships specific to certain subsets of domains. To operationalize this idea, we propose a subset-conditioned invariance objective that aligns class-conditional feature distributions only across domain pairs selected by routing. This alignment is implemented using an optimal transport formulation weighted by expert-specific routing mass, which induces soft domain assignments. To encourage meaningful decomposition, we introduce a diversity regularizer across experts, along with routing objectives that promote both confident (instance-level sparse) and balanced (dataset-level uniform) expert utilization. We evaluate our approach on standard DG benchmarks and demonstrate consistent improvements over strong baselines. Further analysis shows that the learned routing patterns correspond to meaningful domain groupings, supporting the hypothesis that invariant structure is inherently subset-dependent rather than globally shared. Our main contributions are as follows. • We identify a fundamental limitation of global invariance in DG and show that enforcing invariance across increasing numbers of domains can reduce predictive information. Based on this insight, we introduce subset-shared invariance , which reframes DG as learning structured, subset-dependent invariances rather than a single globally invariant representation. • We propose a routing-based MoE framework that learns where invariance should hold. By using routing to induce a data-driven decomposition of domain relationships, our method enforces invariance selectively across domain subsets, enabling adaptive composition of invariant components. • We demonstrate that the proposed approach improves out-of-domain generalization on standard DG benchmarks, particularly under heterogeneous domain shifts, and provides robustness as the number of source domains increases. 2 Related Work Domain Generalization and Invariant Representation Learning. DG is commonly approached by learning representations that are invariant across domains, typically by enforcing the conditional distribution P(Z|Y) to be consistent across environments [ 46 , 18 , 42 ] . This principle underlies a wide range of methods, including feature alignment approaches [ 57 , 51 , 39 ] and invariant learning frameworks such as IRM and its variants [ 1 , 53 , 56 ] . Subsequent work has explored different mechanisms to approximate invariance, including data augmentation [ 54 , 73 ] , episodic and meta-learning strategies [ 2 , 17 ] , and self-supervised learning [ 6 , 43 ] . More recent approaches refine invariance through causal reasoning [ 44 , 43 , 69 , 27 , 36 ] , disentangled representations [ 72 , 11 , 50 ] , and indirect alignment objectives [ 63 ] . Despite their differences, these approaches largely share the assumption that a single representation can be made invariant across all domains. In contrast, we argue that invariant structure is often only partially shared, and should be modeled at the level