Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers
Mana Sakai, Masaaki Imaizumi · 2026
Read Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers by Mana Sakai, Masaaki Imaizumi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07297.txt.
Introduction
Mana Sakai and Masaaki Imaizumi derive generalization bounds for trained transformers that adapt to each model's own spectral shape. Earlier complexity bounds imposed fixed norm limits in advance and could grow exponentially with depth. Their post hoc bounds instead use spectral quantities of the actual trained weight matrices, layer by layer.
The spectral indices can be chosen after training, separately for each matrix type and layer, trading complexity against dimension and depth factors. Proxy comparisons suggest their complexity factors grow more slowly with depth and width than norm-based ones. The work offers a complexity-based view of how trained transformers' spectra relate to generalization.
Catalog record
| CREATOR | Mana Sakai, Masaaki Imaizumi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 147 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07297
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers Mana Sakai, Masaaki Imaizumi Abstract Understanding why trained Transformers generalize well is a fundamental problem in modern machine learning theory, and complexity-based generalization bounds provide a principled way to study this question. While existing norm-based bounds for Transformers remove the explicit polynomial dependence on the hidden dimension, they typically impose fixed norm constraints specified a priori and can exhibit unfavorable exponential dependence on depth. In this paper, we derive spectrum-adaptive post hoc generalization bounds for multi-layer Transformers. Under layerwise spectral norm control, the bounds are expressed in terms of layerwise Schatten quantities of the query-key, value, and feedforward weight matrices. Since the Schatten indices need not be fixed a priori and can instead be selected after training, separately for each matrix type and layer, the bounds adaptively trade off spectral complexity against the dimension- and depth-dependent factors according to the learned singular-value profiles. Empirical comparisons of BERT-adapted proxies for the leading complexity factors suggest that the proxies induced by our bounds grow more slowly with depth and hidden dimension than the corresponding norm-based proxies. Overall, our results provide a complexity-based perspective on how the spectral structure of trained Transformers is reflected in generalization analyses. 1 The University of Tokyo 2 RIKEN Center for Advanced Intelligence Project 3 Kyoto University 1 Introduction Transformers have become a central architecture of modern machine learning since their introduction by 27 . Their success across language ( 7 ; 5 ; 6 ) , vision ( 8 ) , and many other modalities has made it increasingly important to understand why heavily overparameterized Transformers can still generalize well. Generalization bounds offer one principled route toward this goal, because they can reveal which features of a learned model are statistically relevant. For generalization analysis of Transformers, 10 established norm-based generalization gap bounds for multi-layer Transformers. A notable feature of their bounds is that the dependence on both the hidden dimension and the token length appears only through logarithmic factors. 24 sharpened this work by removing the explicit token-length dependence. More recently, 17 used the offset Rademacher complexity to derive excess risk bounds with an O(1/n) rate, where n is the training sample size. At the same time, these advances expose two difficulties that arise when existing norm-based bounds are used to assess trained deep Transformers. First, the improvement in explicit dimension dependence can come with layerwise propagation factors that can scale as C^{L} in the worst case, where L is the depth and C is determined by Lipschitz constants and spectral norm bounds across layers. This issue is benign for shallow networks or when the propagation constant is at most one, but it can quickly make the bounds loose for deep models. Second, the fixed norm radii appearing in existing bounds are not naturally dimension-independent for large matrices. In particular, the bounds of 10 and 24 are controlled by mixed (2,1) - and (1,1) -norm radii, respectively, where the mixed (\alpha,\beta) -norm is defined by \|W\|_{\alpha,\beta}=[\sum_{j}(\sum_{i}|W_{ij}|^{\alpha})^{\frac{\beta}{\alpha}}]^{\frac{1}{\beta}} . Such quantities can remain small for matrices that are sparse in the corresponding coordinatewise sense, but they typically grow with the hidden dimension. A further motivation comes from the observed heterogeneity of trained Transformer weights. Recent compression and spectral analyses suggest that different sublayers can have markedly different singular-value profiles: attention-related matrices often exhibit more low-rank or compressible spectra than feedforward-related matrices ( 16 ; 30 ) . This suggests that a useful complexity measure should not impose a single global norm description on all layers and matrix types. Rather, it should be able to adapt to the spectral profile of each weight. This point is especially important for post hoc generalization bounds. In post hoc bounds, the high-probability event is specified independently of the trained weights, while the admissible complexity parameters may be chosen after the weights are observed. Such guarantees are useful when the relevant structure of the trained model is not known a priori. In the context of DNNs and CNNs, 14 derived generalization bounds based on Schatten (quasi) norms that hold simultaneously over all choices of layerwise Schatten indices, so that these indices can be selected after training. Here, for a matrix W and a Schatten index p\in(0,2] , the Schatten p (quasi) norm is defined by \|W\|_{\mathrm{s},p}=(\sum_{i}\sigma_{i}(W)^{p})^{1/p} , where \sigma_{i}(W) is the i -th singular value. For Transformers, post hoc adaptivity is particularly natural because the spectral profiles of trained matrices may differ substantially across layers. In this paper, we derive post hoc generalization gap bounds for multi-layer Transformers that can adaptively reflect the spectral structure of each learned weight. Specifically, our bounds are based on spectral quantities of the weights: besides the Schatten quantity \|W\|_{\mathrm{s},p}^{p}\ (p\in[0,2]) , with the convention \|W\|_{\mathrm{s},0}^{0}=\operatorname{rank}(W) , we also use the spectral norm \|W\|_{2}=\sigma_{1}(W) . Table 1 summarizes a specialized version of our result along with existing norm-based bounds. A notable feature of our bounds is that the complexity measure, through the Schatten index p , can be selected after training. Table 1 : Leading complexity factor B in simplified generalization gap bounds of the form \tilde{O}(B/\sqrt{n}) . All weights are N\times N and satisfy spectral norm constraints \|W\|_{2}=O(1) . Here L is the depth, n is the sample size, and C denotes the worst-case layerwise propagation factor appearing in existing norm-based bounds. For our post hoc bounds, the displayed expression is the common- p specialization of Theorem 3.1 ; the theorem itself allows a separate p for every matrix type and layer. While our bounds and the bounds of 10 depend on the token length logarithmically, the bounds of 24 are independent of the token length. Bound Assumption Post Hoc Ours (Theorem 3.1 ) \inf_{p}(\|W\|_{\mathrm{s},p}^{p})^{\frac{1}{p+2}}C^{\frac{Lp}{p+2}}L^{\frac{2p+2}{p+2}}N^{\frac{p+1}{p+2}} \surd 10 C_{2,1}C^{L}L^{\frac{3}{2}} \|W\|_{2,1}\leq C_{2,1} 24 C_{1,1}C^{L}L^{\frac{3}{2}} \|W\|_{1,1}\leq C_{1,1} These bounds have several consequences. First, they provide a concrete interpolation between rank-based and norm-based regimes. When p=0 , the Schatten quantity is the rank; when p=2 , it is the squared Frobenius norm. Intermediate values of p describe a soft-rank measure through singular-value decay. Second, this interpolation directly controls the trade-off among spectral complexity, hidden dimension, and depth. Smaller values of p reduce the layerwise propagation factor C^{Lp/(p+2)} , the polynomial depth factor L^{(2p+2)/(p+2)} , and the hidden dimension N^{(p+1)/(p+2)} , whereas larger values of p provide a more Frobenius-like description. Because our bounds take an infimum over p after training, they automatically select the most favorable balance for each learned matrix. Third, using BERT Miniatures checkpoints of 26 , we find that the leading complexity factors suggested by our bounds grow substantially more slowly than the corresponding norm-based complexity factors of 10 as the depth or hidden dimension increases. To obtain these spectrum-adaptive bounds, we derive covering number bounds that retain the dependence on the Schatten indices throughout the Transformer composition. Technically, we develop covering number bounds under layerwise spectral norm and Schatten-quantity constraints, building on the recently developed parametric interpolation ( 15 ; 14 ) . This technique decomposes each weight matrix into a low-rank leading part and a Frobenius-controlled tail, thereby allowing the bounds to interpolate between rank-based and norm-based regimes. The contributions of this paper are summarized as follows. • We prove post hoc generalization gap bounds for multi-layer Transformers under