PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory
Chenyang Wang, Yun Yang · 2026
Read PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory by Chenyang Wang, Yun Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17219.txt.
Introduction
Wang and Yang derive clear PAC-Bayes bounds for Gibbs posteriors — spreads over knobs got by tilting a prior with observed risk. Unlike old worst-case bounds, their analysis yields mean risk bounds for big models, tuned to data structure and model depth. The bound has an edge sum over knob space, studied with singular learning theory to map risk. Uses in low-rank matrix filling and ReLU network fitting and sorting show bounds tractable and clearly tighter than old bounds.
Catalog record
| CREATOR | Chenyang Wang, Yun Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111673 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17219
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory Chenyang Wang, Yun Yang Abstract We derive explicit non-asymptotic PAC-Bayes generalization bounds for Gibbs posteriors, that is, data-dependent distributions over model parameters obtained by exponentially tilting a prior with the empirical risk. Unlike classical worst-case complexity bounds based on uniform laws of large numbers, which require explicit control of the model space in terms of metric entropy (integrals), our analysis yields posterior-averaged risk bounds that can be applied to overparameterized models and adapt to the data structure and the intrinsic model complexity. The bound involves a marginal-type integral over the parameter space, which we analyze using tools from singular learning theory to obtain explicit and practically meaningful characterizations of the posterior risk. Applications to low-rank matrix completion and ReLU neural network regression and classification show that the resulting bounds are analytically tractable and substantially tighter than classical complexity-based bounds. Our results highlight the potential of PAC-Bayes analysis for precise finite-sample generalization guarantees in modern overparameterized and singular models. Keywords: Empirical risk minimization, Generalization bound, Gibbs posterior, Overparameterized model, PAC-Bayes, Singular model 1 Introduction Understanding and controlling the generalization error of learning algorithms remains a central goal in statistical learning theory ( 30 ; 48 ) . Classical bounds based on uniform laws of large numbers, such as those involving VC dimension, Rademacher complexity, or metric entropy control of the model class ( 18 ; 46 ; 8 ) , provide worst-case guarantees on the excess risk of empirical minimizers. While these tools have offered valuable insights, they often result in loose or vacuous bounds in modern overparameterized settings, especially when the model space is high-dimensional or weakly regularized. Indeed, recent theoretical work has shown that worst-case, uniform-convergence-based analyses may be fundamentally insufficient to explain the generalization behavior observed in deep learning ( 28 ; 25 ) , where models with far more parameters than data points can still achieve remarkable performance. An alternative and increasingly prominent line of work uses Bayesian and PAC-Bayes frameworks ( 35 ; 43 ; 32 ) for averaged-case analysis, which replace pointwise guarantees with risk bounds averaged over posterior distributions. These bounds have several practical advantages: they are often more data-dependent, avoid explicit covering number arguments ( 4 ) , and can provide tighter control of generalization error ( 53 ; 25 ; 24 ) . In particular, Bayes-type bounds adapt to model complexity by balancing empirical fit and prior regularity through posterior averaging, without requiring control of the complexity of the entire model class. In this work, we focus on the Gibbs posterior ( 51 ; 20 ; 10 ) , which is a loss-driven posterior distribution that generalizes the traditional Bayesian update by replacing the negative log-likelihood with an empirical risk function induced by a loss. This formulation allows for greater flexibility in modeling and improved robustness to model misspecification, while still enabling principled uncertainty quantification. Recent advances have established PAC-Bayes bounds for Gibbs posteriors under Bernstein-type conditions ( 4 ) , and corresponding posterior contraction rates have been derived by 36 , enabling finite-sample generalization guarantees even in the absence of an explicit likelihood structure. Building on this line of work, we derive an explicit finite-sample PAC-Bayes generalization bound for Gibbs posteriors. In contrast to standard PAC-Bayes inequalities, which separately balance the empirical risk average and a complexity term expressed through the Kullback–Leibler (KL) divergence, our analysis combines these two components into a single bound involving a marginal-like integral over the parameter space. This quantity plays a role analogous to a partition function (or statistical evidence) in Bayesian inference. Through this integration effect, Bayesian methods exhibit intrinsic adaptivity to model complexity, providing a Bayesian counterpart to generalization control beyond worst-case complexity measures derived from uniform laws of large numbers. In regular models, this integral admits a classical Laplace approximation ( 17 ) , recovering the Bayesian Information Criterion ( 33 , BIC,) , which is known to balance goodness of fit and model complexity. Consequently, our formulation reveals that selecting a model that optimally trades off data fit and complexity is equivalent to selecting the model that achieves the tightest PAC-Bayes generalization bound. To analyze this integral, we use tools from singular learning theory developed by 42 , which builds on algebraic geometry to study the asymptotics of marginal likelihoods in non-regular models. In regular models, the leading term in the expansion of the marginal likelihood is d/2 , where d is the number of parameters. In singular models, the leading term is the real log canonical threshold (RLCT), denoted by \lambda . In highly non-identifiable or overparameterized models, \lambda can be much smaller than d/2 and may take fractional values rather than half-integers. This reflects the geometry of the parameterization: the set of minimizers of the population risk often forms a manifold with many flat directions along which the risk is constant. The RLCT effectively counts only the non-flat directions, since parameters along flat directions do not contribute to the asymptotic expansion of the associated singular integral, resulting in sharper generalization bounds. However, most existing results in singular learning theory ( 42 , e.g.) are asymptotic in nature and focus on fixed-dimension, increasing-sample-size regimes, which are not directly applicable to analyzing overparameterized models. They also typically rely on asymptotic expansions under abstract regularity conditions that are often difficult to verify in practice. In contrast, we develop a new non-asymptotic framework in which a sub-exponential loss assumption enables a two-sided PAC-Bayes control between the empirical and population risks. This bypasses the need for explicit control of metric entropy (integrals) or any worst-case uniform convergence arguments, and leads directly to fast-rate PAC-Bayes bounds that depend solely on the RLCT \lambda determined by the model and the loss function. This approach avoids many technical conditions required in asymptotic analyses and yields sharper finite-sample bounds on the expected posterior risk. We illustrate our theory through two canonical examples: low-rank matrix completion ( 3 ; 1 ; 47 ) and deep ReLU neural networks ( 29 ) under squared error loss, as well as classification with logistic loss ( 13 ; 52 ) . In particular, our analysis of the RLCT for matrix completion using blow-up techniques from algebraic geometry also provides insight into the geometry of the model, revealing how the model configurations can be divided into different regimes corresponding to distinct singularity patterns, each associated with a different RLCT. In all cases, the resulting singular integrals admit either closed-form analyses or well-controlled analytical upper bounds, yielding explicit and interpretable generalization bounds that remain tight even in highly non-regular settings. This illustrates the potential of our framework for analyzing complex and overparameterized models, where traditional techniques may be less effective. 2 Preliminaries and Problem Formulation In this section, we set up the notation, review empirical risk minimization (ERM), Gibbs posteriors, PAC-Bayes bounds, and singular learning theory, and reveal the connections among them that motivate our main theoretical results. 2.1 Empirical risk minimization Let Z=(X,Y)\sim\mathbb{P} be a generic data point in \mathcal{X}\times\mathcal{Y} , and let \Theta be a compact parameter space. We consider learning problems specified by a loss function \ell_{\theta}(Z):\mathcal{X}\times\mathcal{Y}\to\mathbb{R} , which quantifies the discrepancy between the prediction under parameter \theta and the observed response. Examples