How Does Preconditioning Guide Feature Learning in Deep Neural Networks?
Kotaro Yoshida, Atsushi Nitanda · 2025
Read How Does Preconditioning Guide Feature Learning in Deep Neural Networks? by Kotaro Yoshida, Atsushi Nitanda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25637.txt.
Introduction
Kotaro Yoshida and Atsushi Nitanda study how preconditioning shapes feature learning and generalization in neural networks. Preconditioners are usually judged by faster training, while their effect on expected risk is less understood. The authors show that the model's input information flows only through the Gram matrix defined by the preconditioner's metric, creating a controllable spectral bias.
Using powers of the input covariance matrix as the preconditioner, they prove results for a single rectified linear unit. Raising the power shifts the learned feature toward higher-variance directions while final population loss stays the same. Experiments on noise robustness, out-of-distribution generalization, and transfer show generalization improves when the bias matches task-relevant features.
Catalog record
| CREATOR | Kotaro Yoshida, Atsushi Nitanda |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-25637
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Does Preconditioning Guide Feature Learning in Deep Neural Networks? Kotaro Yoshida, Atsushi Nitanda Abstract Preconditioning is widely used in machine learning to accelerate convergence on the empirical risk, yet its role on the expected risk remains underexplored. In this work, we investigate how preconditioning affects feature learning and generalization performance. We first show that the input information available to the model is conveyed solely through the Gram matrix defined by the preconditioner’s metric, thereby inducing a controllable spectral bias on feature learning. Concretely, instantiating the preconditioner as the p p -th power of the input covariance matrix and within a single-index teacher model, we prove that in generalization, the exponent p p and the alignment between the teacher and the input spectrum are crucial factors. We further investigate how the interplay between these factors influences feature learning from three complementary perspectives: (i) Robustness to noise, (ii) Out-of-distribution generalization, and (iii) Forward knowledge transfer. Our results indicate that the learned feature representations closely mirror the spectral bias introduced by the preconditioner—favoring components that are emphasized and exhibiting reduced sensitivity to those that are suppressed. Crucially, we demonstrate that generalization is significantly enhanced when this spectral bias is aligned with that of the teacher. 1 Introduction In machine learning, preconditioning has traditionally aimed to accelerate convergence and reduce computational costs during training. By utilizing second-order information, such as the Hessian ( LeCun et al. 2002 ) or the Fisher Information Matrix (FIM) ( Amari 1998 ) , researchers have sought to navigate ill-conditioned loss landscapes ( Li et al. 2018 ) more efficiently. Yet, the ultimate objective extends beyond convergence speed to achieving strong generalization on unseen data. This generalization fundamentally hinges on the nature of the feature representations learned during training ( Bengio et al. 2013 ; Yosinski et al. 2014 ; LeCun et al. 2015 ) . Although much of the literature has focused on accelerating convergence on the empirical risk, the impact on the expected-risk performance has often been underexplored ( Dauphin et al. 2014 ; Keskar et al. 2016 ; Wilson et al. 2017 ) . This motivates the question: how does preconditioning affect solution quality—namely, what kinds of features are captured and how they contribute to generalization ? The answer to this question remains controversial and has yet to be systematically established. One perspective suggests that preconditioning, particularly with second-order methods, can be detrimental. This view is supported by arguments that such methods converge to sharp solutions that generalize poorly ( Keskar et al. 2016 ; Shin et al. 2025 ) , or that they are equivalent to data whitening, which can discard useful information ( Wadia et al. 2021 ) . An opposing view, however, highlights the benefits, showing that preconditioning can enhance robustness to label noise ( Amari et al. 2021 ) , stabilize feature learning ( Ishikawa & Karakida 2024 ) , and better handle anisotropic data ( Zhang et al. 2025 ) . It is also argued, from a middle-ground perspective and aligning with the No Free Lunch theorem in optimization ( Wolpert & Macready 1997 ) , that the effectiveness of a preconditioner is not universal, but rather depends on the alignment of its inherent inductive bias with the specific structure of the task ( Amari et al. 2021 ) . Despite these insights, a systematic understanding of the interplay between preconditioning and feature learning in deep learning is still lacking. In this paper, we aim to systematize the relationship between preconditioning and generalization in deep learning. First, we extend the arguments of Wadia et al. 2021 to a general preconditioning framework. We demonstrate that neuron-wise preconditioning in the first layer determines the similarity metric of the Gram matrix, which uniquely conveys input data information to the model during both training and inference. In other words, the model can only access input information through the Gram matrix defined by the preconditioner’s metric; the preconditioner directly shapes this similarity space, thereby directly influencing learning and inference. Next, we instantiate this insight by approximating the first-layer preconditioner as the p -th power of the input covariance matrix. We show that the exponent p determines the similarity geometry of the Gram matrix, selectively emphasizing features with specific variance levels. Specifically, in this geometry, a larger p makes high-variance features more dominant, while a smaller p emphasizes low-variance components. Consequently, by considering a single-index teacher model, we conclude that, for a fixed input spectrum, test-time generalization is only governed by both the preconditioner’s power p and the degree of alignment between the teacher and the input spectrum, as well as the level of label noise. We further elucidate how the interplay between the preconditioner’s power p and the alignment between the teacher and the input spectrum governs model generalization, examining this relationship from three perspectives: (i) Robustness to noise. We construct synthetic datasets where a teacher model is aligned with either high- or low-variance components. By training a two-layer MLP with preconditioners based on both the exact covariance matrix and an approximate Hessian, we find that the model becomes sensitive to the variance components emphasized by p , and that the value of p yielding the most robust generalization is the one that matches the variance components to which the teacher is aligned. (ii) Out-of-distribution (OOD) generalization. The trend observed in (i) becomes particularly salient under a correlation shift, where the data contains both invariant and spurious features. We find that OOD generalization is improved when the power p is chosen to emphasize the variance components associated with invariant features while suppressing those associated with spurious features. This is confirmed by comparing various optimizers on an MNIST-based task where small-variance Gaussian noise is added. (iii) Forward knowledge transfer. We connect our framework to transfer and continual learning, where leveraging knowledge from past tasks is crucial. We argue that using an optimizer with a strong spectral bias on a source task can inadvertently discard information vital for subsequent tasks, thereby degrading transfer/continual learning performance. Therefore, we show that setting p=-1 , which transparently incorporates all spectral components during learning on the source task, is preferable for improving transferability to future tasks. We confirm this phenomenon under the same experimental settings as those used in (i). The code is available at https://github.com/katoro8989/preconditioning-feature-learning . In summary, our main contributions are as follows: • We provide a theoretical result: when training deep models with preconditioning, all input information that can affect feature learning is extracted solely through the Gram matrix defined by the similarity geometry of the preconditioning matrix (Sec. 3.1 ). • We instantiate this result to preconditioning based on the p -th power of the input covariance matrix and show that, in the induced Gram matrix, larger p gives greater influence to high eigenvalue components whereas smaller p gives greater influence to low eigenvalue components. Under a single-index teacher model, when the data spectrum is fixed, the model’s generalization is determined only by p and the alignment between the teacher and the input spectrum (and label noise) (Secs. 3.2 and 3.3 ). • We further examine how the relationship between the preconditioner power p and the alignment between the teacher and the input spectrum impacts generalization through