Revisiting the Volume Hypothesis

Ari Pakman, Lior Kreimer, Yakir Berchenko · 2026

Read Revisiting the Volume Hypothesis by Ari Pakman, Lior Kreimer, Yakir Berchenko (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31282.txt.

Introduction

Pakman and two colleagues revisit the volume hypothesis for why big neural networks generalise well. The idea says that, among weight settings with low training loss, basins leading to good generalisation are simply much larger, so gradient descent is likely to land in one. Recent tests seemed to clash: random sampling gave poor results while density estimates backed the hypothesis.

The team note those tests used different dataset sizes and probe the middle ground. With a replica exchange Wang-Landau method they estimate the joint density of training and test accuracy in binary networks. Across architectures and datasets, the edge of gradient learning over random sampling shrinks as training data grows, which resolves the apparent paradox without discarding either earlier finding.

Catalog record

CREATORAri Pakman, Lior Kreimer, Yakir Berchenko
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31282

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revisiting the Volume Hypothesis Ari Pakman, Lior Kreimer, Yakir Berchenko Abstract Modern deep neural networks often contain far more parameters than needed to fit their training data, yet they achieve impressive generalization. A common explanation for this success is the implicit bias of stochastic gradient descent (SGD). An alternative volume hypothesis posits that, within low training-loss regions, loss-landscape basins leading to strong generalization occupy much larger regions of weight space than basins that generalize poorly, and therefore SGD is simply more likely to land in the former. Recent experimental explorations of this idea present seemingly contradictory results. While in one set of experiments randomly sampling the network weights until achieving zero training error yielded poor generalization, molecular dynamics density estimates supported the volume hypothesis. We observe that these experiments were performed at different dataset size regimes, and explore an intermediate regime using the Replica Exchange Wang–Landau algorithm to estimate the joint density of states over training and test accuracies in binary networks. Across several architectures and datasets, we show that the generalization advantage of gradient learning over random sampling training generally diminishes as the training data size grows, suggesting a resolution of the paradox. 1 Introduction From the perspective of classical learning theory ( Shalev-Shwartz & Ben-David 2014 ) , it is surprising that modern neural networks generalize so well to new data. These models usually contain many more parameters than are necessary to fit the training set, and making them even larger tends to improve rather than harm generalization ( Hestness et al. 2017 ) . In an over-parameterized network, countless parameter configurations can drive the training loss to zero—some generalize, others do not. Yet, for reasons that remain unclear, stochastic gradient descent (SGD) routinely lands on parameter configurations that do generalize ( Zhang et al. 2017 ) . The prevailing view attributes this success to an implicit bias that SGD introduces when it trains an over-parameterized model ( Soudry et al. 2018 ; Gunasekar et al. 2018 ; Arora et al. 2019 ; Vardi 2023 ) . However, a recent work by Chiang et al. 2022 proposed a different explanation, the volume hypothesis , according to which within the low training error regions of the weight space, strong generalization regions simply occupy a much larger volume than those that generalize poorly. This idea echoes related proposals in ( Pérez et al. 2019 ; Berchenko 2024 ) and has been explored in several theoretical works in simplified cases ( Hanin & Zlokapa 2023 ; Buzaglo et al. 2024 ; Harel et al. 2024 ; Alexander et al. 2025 ) . If correct, one could argue that architectural bias is the main driver of generalization, while SGD’s implicit bias is secondary. This claim has been recently tested through two different approaches, with opposite outcomes. On the one hand, the work by Peleg & Hein 2024 used a Guess and Check (G&C) procedure: randomly sampling weight values until achieving zero training error. The resulting networks have worse generalization error than SGD. 1 1 1 Peleg & Hein 2024 argue that earlier claims to the contrary by Chiang et al. 2022 can be explained away by proper initialization and loss normalization. On the other hand, the work by Yang et al. 2026 uses molecular dynamics techniques to estimate the density of the network parameters as a function of (i) the training loss and (ii) the generalization loss (for regression) or accuracy (for classification). For low training loss, in many classification and regression models the density has a peak at generalization values similar or better than those reached by highly-optimized SGD. This implies the validity of the volume hypothesis, dubbed high-entropy advantage by Yang et al. 2026 , though in one of their examples the phenomenon disappears for wide enough networks. In order to reconcile these diverging results, we note that the experiments reported in Peleg & Hein 2024 were restricted to binary classification tasks in the low sample regime (up to 32 training samples). Extending G&C to multiclass classification or bigger training datasets is unfeasible due to the high computational cost, which increases quickly with training sample size and the number of classification categories. On the other hand, the experiments in Yang et al. 2026 were performed in the high sample regime. This suggests that the validity of the volume hypothesis might depend strongly on the data size. In this work we explore this idea using the Wang-Landau algorithm ( Wang & Landau 2001 ) , a technique developed in the statistical physics community to compute the probability density of the energy or other macroscopic quantities in systems in which this density can vary across many orders of magnitude. In our case, we use it to estimate the joint density of states g(A,Q) over training accuracy A and test accuracy Q in selected ranges of interest, for multiclass classification networks and training datasets of up to 600 samples. In three different architectures and two datasets, we find the generalization advantage of SGD over random sampling generally diminishes as the training data size grows. This result bridges the different results previously reported. Our results demonstrate a data-dependent transition in which optimization-induced bias dominates in small-data regimes, while architectural volume effects emerge and concentrate with increasing data. This clarifies the respective roles of optimization, architecture, and data in overparameterized generalization. 2 Related works Wang-Landau in machine learning. An estimation of the density of states in binary neural networks using the Wang-Landau method was recently performed in ( Mele et al. 2025 ) . However, this work only considered the density of the training error, and thus does not yield insights on generalization performance. Another recent use of the Wang-Landau method in machine learning is the work ( Liu et al. 2023 ) , which computed the density of output values of a network over the input space, for fixed, previously-trained weights. The volume hypothesis was recently explored by Yang et al. 2026 using a variant of the Wang-Landau method that combines molecular dynamics with non-parametric density estimation. The same technique was recently applied in ( Zhang et al. 2025 ) to the grokking phenomenon. Overparameterization and generalization. The question of why deep neural networks generalize well despite severe overparameterization has been investigated for several decades from a variety of theoretical and empirical perspectives ( Wolpert 1995 ; Bartlett & Mendelson 2001 ; Hoffer et al. 2017 ; Jakubovitz et al. 2019 ) . Classical learning theory relates generalization to capacity control via complexity measures such as the VC dimension ( Vapnik & Chervonenkis 1971 ) , suggesting that highly expressive models should overfit ( Shalev-Shwartz & Ben-David 2014 ; Hastie et al. 2001 ) . From this viewpoint, networks with enough parameters to interpolate arbitrary labels would be expected to generalize poorly. Empirically, however, modern neural networks defy this prediction. Even models capable of perfectly fitting the training data often exhibit strong test performance ( Haeffele & Vidal 2017 ; Nguyen & Hein 2018 ) , and increasing model size can further improve generalization ( Belkin et al. 2019 ; Neal et al. 2019 ; Bartlett et al. 2020 ) . A striking demonstration by Zhang et al. 2017 showed that convolutional networks can

Read the complete text