The Golden Subspace: Where Efficiency Meets Generalization in Continual Test-Time Adaptation
Guannan Lai, Da-Wei Zhou, Zhenguo Li, Han-Jia Ye · 2026
Read The Golden Subspace: Where Efficiency Meets Generalization in Continual Test-Time Adaptation by Guannan Lai, Da-Wei Zhou, Zhenguo Li, Han-Jia Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21928.txt.
Introduction
Lai and co-authors hunt for the golden subspace, the smallest feature set a model must update to adapt at test time. Continual test-time adaptation faces shifting, unlabeled data and no source data. Updating many parameters adapts well but slows online use. They prove it exists for one-step adaptation and equals the classifier's row space. Their GOLD method tracks it with an outer-product proxy and learns a small scaling vector. Tests on classification and segmentation, including driving scenes, led on speed, stability, and score.
Catalog record
| CREATOR | Guannan Lai, Da-Wei Zhou, Zhenguo Li, Han-Jia Ye |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59836 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21928
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Golden Subspace: Where Efficiency Meets Generalization in Continual Test-Time Adaptation Guannan Lai, Da-Wei Zhou, Zhenguo Li, Han-Jia Ye Abstract Continual Test-Time Adaptation (CTTA) aims to enable models to adapt online to unlabeled data streams under distribution shift without accessing source data. Existing CTTA methods face an efficiency–generalization trade-off: updating more parameters improves adaptation but severely reduces online inference efficiency. An ideal solution is to achieve comparable adaptation with minimal feature updates; we call this minimal subspace the golden subspace . We prove its existence in a single-step adaptation setting and show that it coincides with the row space of the pretrained classifier. To enable online maintenance of this subspace, we introduce the sample-wise Average Gradient Outer Product (AGOP) as an efficient proxy for estimating the classifier weights without retraining. Building on these insights, we propose Guided Online Low-rank Directional adaptation (GOLD) , which uses a lightweight adapter to project features onto the golden subspace and learns a compact scaling vector while the subspace is dynamically updated via AGOP. Extensive experiments on classification and segmentation benchmarks, including autonomous-driving scenarios, demonstrate that GOLD attains superior efficiency, stability, and overall performance. Our code is available at https://github.com/AIGNLAI/GOLD . 2 2 footnotetext: Correspondence to: Han-Jia Ye ([email protected]) 1 Introduction In real-world applications, models are often required to perform inference under continuously shifting data distributions 4 ; 15 ; 21 , such as weather and illumination changes in autonomous driving, scene variations in video stream analysis, and device heterogeneity in medical imaging 19 ; 50 ; 49 . These scenarios demand models that can adapt online during test time to unknown and gradually changing target domains, without re-accessing source data or performing offline retraining. This problem setting, known as Continual Test-Time Adaptation (CTTA) , aims to enable models to maintain robust performance in dynamic environments 44 ; 12 ; 36 . (a) Efficiency and Accuracy (b) Generalization Figure 1 : Results on the CIFAR100-C dataset. Left: performance–efficiency distribution of evaluated methods (TCA 33 is closed-source and shown as a line). Right: accuracy over the continuous test-time adaptation process; shaded background bands indicate periods of domain shift. In the golden-shaded intervals, prior methods exhibit significant performance degradation, whereas our method maintains generalization. In practical deployments, CTTA faces a trade-off between efficiency and generalization: better generalization typically requires more complex models and more intensive computation, reducing run-time efficiency. Existing approaches typically update model parameters at test time using self-supervised objectives, entropy minimization, or adaptive batch-normalization statistics 27 ; 5 . While such updates can provide short-term gains, they substantially increase computational cost and tend to amplify pseudo-label noise and induce parameter drift, undermining the generalization of the continual adaptation process and ultimately degrading long-term performance 10 ; 40 ; 45 . As shown in Fig. 1(b) , when a new domain arrives, existing methods fail to adapt quickly and exhibit sudden performance degradation, indicating an inability to preserve generalization during adaptation. Ideally, our goal is to achieve the desired change in model output within the feature subspace (to ensure generalization) while keeping the update magnitude minimal (to ensure efficiency). We define this subspace as the Golden Subspace . We first consider a single-step adaptation scenario, derive its analytical formulation, and verify the existence of the golden subspace. Our analysis reveals that this subspace is essentially composed of the row-space of the classifier weights, which can be obtained via eigenvalue decomposition of the classifier weight matrix. However , a new challenge arises: the classifier weights encode predominantly source-domain information, and continuously updating them at test time reintroduces the same problems of high computational cost and structural degradation. We therefore ask: how can the classifier be endowed with target-domain information without retraining its weight? Motivated by recent advances on Average Gradient Outer Product (AGOP) representations 39 ; 3 ; 30 ; 1 ; 2 , we find that AGOP computed from high-confidence samples provides a faithful online proxy for the classifier’s parameter inner-product structure. Specifically, the golden subspace estimated via AGOP converges over time to the subspace obtained by single-step least-norm adaptation, which guarantees that models built on this estimator retain strong generalization . Empirically, AGOP matrices exhibit low effective rank, implying that the golden subspace is low-rank in practice and therefore amenable to highly efficient online maintenance and adaptation. Building on these insights, we propose Guided Online Low-rank Directional adaptation (GOLD) , a method for efficient continuous test-time adaptation. GOLD maintains a lightweight matrix, which is initialized from the classifier weight inner-products and is updated online using the AGOP computed from high-confidence test samples. We periodically perform an eigendecomposition of it to extract the current low-rank golden subspace . Features from a frozen backbone are projected onto this subspace, and a compact scaling vector is learned to re-scale the projected coordinates for adaptation. This design introduces only a small number of external parameters and updates a very limited internal parameter subset, thereby intrinsically constraining parameter drift while providing sufficient capacity to refine outputs. As shown in Figure 1 , GOLD attains the best performance with minimal adaptation cost ( Figure 1(a) ) and preserves high generalization ability across domain shifts ( Figure 1(b) ). Our main contributions are summarized as follows: • We formalize the golden subspace as the minimal feature-update subspace induced by the classifier and show that it admits a low-rank characterization. We further develop an AGOP-based online estimator that tracks this subspace during test-time adaptation. • We introduce GOLD, a source-free online adaptation framework that projects features onto the golden subspace and learns a lightweight scaling vector. GOLD achieves efficient adaptation with minimal updates. • Extensive experiments on classification and segmentation benchmarks demonstrate that GOLD achieves SOTA performance with a minimal computational footprint. 2 Related Work Continual Test-Time Adaptation (CTTA) 26 ; 16 extends test-time adaptation to dynamic environments 22 ; 23 , where a source model must adapt continuously to a stream of unlabeled and non-stationary target data without revisiting source samples 29 ; 32 . The pioneering CoTTA 44 introduced a teacher student framework with weight averaging and stochastic weight restoration to alleviate error accumulation and forgetting. Building on this idea, PETAL 5 proposed a data-driven parameter recovery mechanism that regularizes model updates toward the source parameter configuration to enhance robustness. RMT 13 and SANTA 8 enforced feature-level consistency with the source model through contrastive objectives, while DSS 45 improved reliability by filtering unsafe pseudo-labels during adaptation. More recently, TCA 33 maintained inter-class stability by preserving topological consistency among class representations during domain shifts. Despite these advances, most existing methods still rely on global or feature-level updates of the entire network 41 ; 55 ; 11 , which inevitably leads to a trade-off between efficiency and generalization. 3 Preliminary 3.1 Problem Definition We study the CTTA setting 47 ; 25 , where a model f_{\theta}=h_{\psi}\circ g_{\phi} is pretrained on a labeled source domain \mathcal{D}_{\mathrm{src}}=\{(x_{i}^{\mathrm{src}},y_{i}^{\mathrm{src}})\}_{i=1}^{N_{\mathrm{src}}} and deployed to an unlabeled target data stream \mathcal{D}_{T}=\{\mathcal{X}_{1},\mathcal{X}_{2},\ldots,\mathcal{X}_{T}\} . Each batch \mathcal{X}_{t}=\{x_{i}^{(t)}\}_{i=1}^{N_{t}} is drawn from a target distribution p_{t}(x) that may evolve over time, i.e., p_{t}(x)\neq p_{t+1}(x) . During deployment, the model encounters the data stream sequentially. At each step t , it receives the current batch \mathcal{X}_{t} , produces predictions \hat{y}^{(t)}=g_{\theta_{t}}(\mathcal{X}_{t}) , and performs online adaptation