Estimating the Probability of Sampling a Trained Neural Network at Random

Adam Scherlis, Nora Belrose · 2025

Read Estimating the Probability of Sampling a Trained Neural Network at Random by Adam Scherlis, Nora Belrose (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18812.txt.

Introduction

Adam Scherlis and Nora Belrose estimate how likely a trained neural network is to appear by pure chance in a 2025 study. Their method measures the local volume around a chosen network: the size of the neighborhood in parameter space whose networks behave much the same. A fast existing estimator gives only a lower bound on that volume in many cases, which they improve with an importance sampling scheme guided by gradient information.

Volumes under bell-shaped and uniform measures are both covered. Knowing how special a trained solution is, in plain volume terms, feeds debates about why learning finds good networks at all.

Catalog record

CREATORAdam Scherlis, Nora Belrose
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18812

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Estimating the Probability of Sampling a Trained Neural Network at Random Adam Scherlis, Nora Belrose Abstract We present and analyze an algorithm for estimating the size, under a Gaussian or uniform measure, of a localized neighborhood in neural network parameter space with behavior similar to an “anchor” point. We refer to this as the “local volume” of the anchor. We adapt an existing basin-volume estimator, which is very fast but in many cases only provides a lower bound. We show that this lower bound can be improved with an importance-sampling method using gradient information that is already provided by popular optimizers. The negative logarithm of local volume can also be interpreted as a measure of the anchor network’s information content. As expected for a measure of complexity, this quantity increases during language model training. We find that overfit, badly-generalizing neighborhoods are smaller, indicating a more complex learned behavior. This smaller volume can also be interpreted in an MDL sense as suboptimal compression. Our results are consistent with a picture of generalization we call the “volume hypothesis”: that neural net training produces good generalization primarily because the architecture gives simple functions more volume in parameter space, and the optimizer samples from the low-loss manifold in a volume-sensitive way. We believe that fast local-volume estimators are a promising practical metric of network complexity and architectural inductive bias for interpretability purposes. 1 Introduction There is a long line of research which finds that flat minima in a neural network parameter space, defined as weight vectors surrounded by large regions “with the property that each weight vector from that region leads to similar small error” generalize better than sharp minima ( Hochreiter & Schmidhuber, 1997 ) . While there are counterexamples to this tendency ( Dinh et al., 2017 ) , it seems to be empirically and theoretically fairly robust, and has inspired the development of optimizers that explicitly search for flatter minima ( Foret et al., 2021 ) . In a related line of work, Chiang et al. (2022) put forward the volume hypothesis , which states that “…the implicit bias of neural networks may arise from the volume disparity of different basins in the loss landscape, with good hypothesis classes occupying larger volumes.” They evaluate simple gradient-free learning algorithms, such as the “Guess & Check” optimizer which randomly samples parameters until it stumbles upon a network that achieves training loss under some threshold, and find that these methods have similar generalization behavior to gradient descent, at least on the very simple tasks they tested. Teney et al. (2024) find that randomly initialized networks represent very simple functions, which would explain the simplicity bias of deep learning if SGD behaves similarly to Guess & Check. Additionally, Mingard et al. (2021) provide evidence that SGD may be an approximate Bayesian sampler, where the prior distribution over functions is equal to the distribution over functions represented by randomly initialized networks. Since networks are usually initialized using a uniform or Gaussian distribution, the Bayesian sampling hypothesis makes similar predictions to the volume hypothesis. Finally, recent work suggests that singular learning theory ( Watanabe, 2009 ) , originally developed to analyze the learning dynamics of overparameterized Bayesian models, can be profitably used to understand deep learning. The primary tool used in this work is the Local Learning Coefficient, a measurement of network complexity ( Hoogland et al., 2024 ; Lau et al., 2024 ) . In this work, we propose an efficient algorithm for estimating the size of a behaviorally-defined neighborhood of parameter space according to a Gaussian or uniform measure. This is equivalent to the usual volume of the neighborhood if the measure is uniform; for a Gaussian measure it has better practical properties but a slightly different interpretation. Each neighborhood is anchored by a reference set of parameters. We refer to this measure as the “local volume” of the anchor, for either the Gaussian or uniform case. If the measure is normalized as a probability distribution, we can interpret the local volume as the probability it assigns to the neighbhorhood. In this case we also refer to the measure as a “prior”. We distinguish two variants of local volume, depending on the behavioral definition used for the neighborhood: • Local loss volume is defined with a loose behavioral constraint (low loss across a dataset) and is simpler to interpret • Local KL volume is defined with a strict behavioral constraint (low KL divergence from the anchor across all inputs) but a looser connection to training loss As an example, we estimate that the probability of randomly sampling the trained Pythia 31M language model from its initialization distribution, within 0.01 nats of KL divergence, is about \mathrm{Pr}(\text{Pythia 31M})\approx\frac{1}{10^{3.6\times 10^{8}}} (1) or one in 1 followed by 360 million zeros. For comparison, there are about 10^{80} atoms in the observable universe, so this is about the same as the probability of correctly guessing a specific atom 4.5 million times in a row. This serves to illustrate the sorts of numbers involved, so that statements about orders of magnitude can be taken in their appropriate context. 2 Local volume Formally, given a weight vector \theta\in\mathbb{R}^{N} , a cost function \mathcal{C}:\mathbb{R}^{N}\rightarrow[0,\infty) , and a threshold \epsilon>0 , we define a neighborhood to be the largest star domain \mathcal{S} anchored at \theta such that \mathcal{C}(\theta^{\prime})<\epsilon for all points \theta^{\prime}\in\mathcal{S} . 1 1 1 This should be thought of as similar to a uniformly low-loss region, in contrast to a “basin” in the sense we use it below: the entire region of parameter space that flows towards a low-loss region. Given a measure \mu (uniform or Gaussian), the local volume of \theta (for a particular \mathcal{C} and \epsilon ) is defined to be the total measure, under \mu , of the corresponding neighborhood. If \mu is normalized as a probability distribution, this is equivalent to the probability of sampling a network inside \mathcal{S} from \mu . While prior work has assumed \mu to be the Lebesgue measure (i.e. volume), we also consider the probability measure used to initialize the network before training, which guarantees that the measure of any neighborhood must be finite. (We have found empirically that some real-world neighborhoods actually have infinite Lebesgue volume, creating difficulties for analysis.) We primarily consider two kinds of cost functions (and correspondingly two kinds of neighborhoods) in this work: 1. Loss neighborhoods: when \mathcal{C}(\theta^{\prime}) is the expected loss of \theta^{\prime} on a dataset of inputs and ground-truth outputs 2. KL neighborhoods: when \mathcal{C}(\theta^{\prime}) is the expected KL divergence \mathbb{E}_{x}[D_{KL}(f(x;\theta)||f(x;\theta^{\prime})] of \theta^{\prime} from \theta , with x being drawn from a dataset of inputs only KL neighborhoods have several practical and theoretical advantages, so we focus on them in this paper: • KL measures how behaviorally different \theta^{\prime} is from \theta , independent of any ground truth labels. This is a much stronger behavioral constraint than loss, and we believe it makes the KL neighborhoods much more compact and therefore tractable to estimate volume for. • KL is always zero at the minimum point \theta^{\prime}=\theta . • KL local volume provides a direct measure of architectural inductive bias, which (as we will argue) corresponds closely to functional complexity. 2 2 2 We can compare

Read the complete text