Scaling Up Temporal Domain Generalization via Temporal Experts Averaging
Aoming Liu, Kevin Miller, Venkatesh Saligrama, Kate Saenko, Boqing Gong, Ser-Nam Lim, Bryan A. Plummer · 2025
Read Scaling Up Temporal Domain Generalization via Temporal Experts Averaging by Aoming Liu, Kevin Miller, Venkatesh Saligrama, Kate Saenko, Boqing Gong, Ser-Nam Lim, Bryan A. Plummer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-26045.txt.
Introduction
Liu and a team including Plummer tackle a drift problem for language models in twenty twenty-five. Language and usage change over time, so a model trained on old data can fail on new data. Past methods tried to predict future model weights, but that is costly for large models. Most thus adjust only the final layer, which limits gains.
Their method, called Temporal Experts Averaging, updates the whole model by averaging weights. First, they fine tune a base model on each time period, with limits on how far weights can move. Next, they weight each expert by how close it may be to future data. Tests on seven benchmarks show large gains over prior methods, with much lower cost.
Catalog record
| CREATOR | Aoming Liu, Kevin Miller, Venkatesh Saligrama, Kate Saenko, Boqing Gong, Ser-Nam Lim, Bryan A. Plummer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-26045
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling Up Temporal Domain Generalization via Temporal Experts Averaging Aoming Liu, Kevin Miller, Venkatesh Saligrama, Kate Saenko, Boqing Gong, Ser-Nam Lim, Bryan A. Plummer Abstract Temporal Domain Generalization (TDG) aims to generalize across temporal distribution shifts, e.g., lexical change over time. Prior work often addresses this by predicting future model weights. However, full model prediction is prohibitively expensive for even reasonably sized models. Thus, recent methods only predict the classifier layer, limiting generalization by failing to adjust other model components. To address this, we propose Temporal Experts Averaging (TEA), a novel and scalable TDG framework that updates the entire model using weight averaging to maximize generalization potential while minimizing computational costs. Our theoretical analysis guides us to two steps that enhance generalization to future domains. First, we create expert models with functional diversity yet parameter similarity by fine-tuning a domain-agnostic base model on individual temporal domains while constraining weight changes. Second, we optimize the bias-variance tradeoff through adaptive averaging coefficients derived from modeling temporal weight trajectories in a principal component subspace. Expert’s contributions are based on their projected proximity to future domains. Extensive experiments across 7 TDG benchmarks, 5 models, and 2 TDG settings shows TEA outperforms prior TDG methods by up to 69% while being up to 60x more efficient 1 1 1 Code: https://github.com/zxcvfd13502/TEA . 1 Introduction Temporal Domain Generalization (TDG) ying Bai et al. 2022 ; Nasery et al. 2021 ; Qin et al. 2022 ; Xie et al. 2024c ; Xie et al. 2024a ; Yong et al. 2023 ; Xie et al. 2024b aims to generalize to unseen future data under temporal distribution shift without retraining the models, as illustrated in Fig. 1 . Unlike traditional Domain Generalization (DG), which lacks target domain information Li et al. 2017a ; Muandet et al. 2013 ; Li et al. 2018a ; Li et al. 2018b , TDG often leverages temporal patterns for prediction to better adapt the models for future domains by predicting model weights, such as forecasting NLP research trends Yao et al. 2022a . However, prior work has scaling limitations. For example, early brute-force methods predict entire models, but have prohibitive computational costs when scaling up the model and dataset size Nasery et al. 2021 ; ying Bai et al. 2022 ; Qin et al. 2022 . As shown in Fig. 2(a) , recent methods improve efficiency by only predicting the classifier Xie et al. 2024c ; Xie et al. 2024b , but sacrifice generalization capabilities from keeping other model components frozen. Thus, these methods struggle to surpass basic ERM baselines when scaling-up benchmarks Yao et al. 2022a ; Lin et al. 2022 . (a) Temporal distribution shift in vehicle appearance. (b) Temporal distribution shift in NLP paper keywords. Figure 1: Examples of temporal domain generalization (TDG) span both (a) vision and (b) language tasks. TDG aims at enabling models trained on historical data to directly generalize to future data without retraining. (a) Classifier-only TDG framework (b) Our Temporal Expert Averaging (TEA) framework Figure 2: TDG framework comparison. (a) Classifier-only TDG Xie et al. 2024c ; Xie et al. 2024b only predicts future classifiers to reduce computational costs in scaled-up scenarios, but limits generalization potential by neglecting other model components. (b) Our Temporal Expert Averaging (TEA) enables higher generalization potential by adjusting the entire model through predicting future averaging coefficients of temporal experts capturing diverse functionalities. The low-dimensional nature of these coefficients ensures TEA’s efficiency in scaled-up scenarios. To address the scaling challenges, we propose Temporal Experts Averaging (TEA), a TDG framework based on weight averaging (WA) that predicts the averaging coefficients of temporal experts for future domains. However, WA methods for DG, e.g., Cha et al. 2021 ; Rame et al. 2022 ; Wortsman et al. 2022 , lack mechanisms to exploit temporal patterns in TDG. We identify two key capabilities required to leverage temporal patterns for WA through a bias-variance-covariance-locality decomposition analysis of generalization error. First, model weights need functional diversity, yet parameter similarity. Second, optimizing averaging coefficients to achieve better bias-variance tradeoffs than uniform averaging. To this end, our TEA satisfies the first criteria by training a domain-agnostic base model on all source domains, followed by constrained incremental fine-tuning on each individual domain. To fulfill the second criteria, we extract principal components from the deviations between expert weights and the base model, creating a low-dimensional subspace to model temporal weight trajectories. This enables forecasting future domain positions and averaging experts based on their projected proximity to the future domain. This enables TEA to temporally-adapt all model parameters with computational costs comparable to standard ERM training, offering higher generalization potential than merely adjusting the classifier. The superiority of TEA is demonstrated through comprehensive evaluation across 7 diverse TDG benchmarks and 5 different models, covering both vision and language tasks. Beyond standard TDG with simultaneous access to all source domains, we also evaluate on Continual Domain Generalization over Temporal Drift (CDGTD) settings, where new domains arrive sequentially. Across this extensive evaluation, TEA consistently achieves new state-of-the-art results, outperforming prior TDG methods by up to 69% while being up to 60x more efficient. Our contributions can be summarized as follows: • We propose TEA, a novel weight-averaging-based TDG framework that efficiently enhances generalization across temporal shifts with broad model/dataset compatibility. • We provide valuable theoretical insights on the under-explored WA-TDG integration, design our method based on these insights, and validate our insights and method through superior generalization performance across various benchmarks. • We enhance TDG evaluation comprehensiveness by addressing both TDG and CDGTD, unlike prior work that typically focused on just one setting. This includes introducing CLEAR-10 and CLEAR-100 ( Lin et al. 2022 ) as new evaluation benchmarks for TDG. 2 Related Work Temporal Domain Generalization (TDG) Ortiz-Jiménez et al. 2019 ; Mancini et al. 2019 ; Wang et al. 2020 ; ying Bai et al. 2022 ; Nasery et al. 2021 ; Zeng et al. 2023 ; Wang et al. 2022 ; Xie et al. 2024c ; Xie et al. 2024a ; Yong et al. 2023 ; Xie et al. 2024b exploits temporal patterns in ordered domains with smooth distribution shifts to enhance generalization to future domains. Early approaches like GI Nasery et al. 2021 and DRAIN ying Bai et al. 2022 predict entire model parameters, but face computational challenges with large-scale models, while recent methods like EvoS Xie et al. 2024c and W-Diff Xie et al. 2024b reduce costs by only adjusting classifiers, potentially limiting generalization. TDG encompasses multiple settings: the original setting with simultaneous access to all source domains, Continual Domain Generalization over Temporal Drift (CDGTD) with sequentially available domains, and Continuous Temporal Domain Generalization (CTDG) for continuously distributed temporal data. We focus on the original TDG and CDGTD settings as CTDG remains impractical for most realistic benchmarks. Domain Adaptation and Generalization. Enabling models to perform well on out-of-distribution (OOD) data has been a crucial challenge in machine learning. Two specific tasks highly relevant to our work are Domain Adaptation (DA) and Domain Generalization (DG). DA methods Saenko et al. 2010 ; Sun et al. 2015 ; Sun and Saenko 2016 ; Gong et al. 2012 ; Tzeng et al. 2017 ; Li et al. 2016 typically adapt models against distribution