The Space Between: On Folding, Symmetries and Sampling

Michal Lewandowski †, Bernhard Heinzl †, Raphael Pisoni †, Bernhard A.Moser †* · 2025

Read The Space Between: On Folding, Symmetries and Sampling by Michal Lewandowski †, Bernhard Heinzl †, Raphael Pisoni †, Bernhard A.Moser †* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08502.txt.

Introduction

Neural networks fold their input space as they learn, and Michal Lewandowski and co-authors present a way to measure that folding in a 2025 paper. Recent work suggests that layers with ReLU activation fold the space step by step. A folding measure based on Hamming distance in the activation space was proposed not long ago, but it covered a narrow class of functions.

The authors generalize the measure to a wider class of activation functions by grouping inputs into equivalence classes. They analyze its math and compute traits, and they give an efficient sampling strategy for its use. Folding values rise with depth when error is low and fall when error grows. A new regularization scheme is outlined that pushes networks toward higher folding.

Catalog record

CREATORMichal Lewandowski †, Bernhard Heinzl †, Raphael Pisoni †, Bernhard A.Moser †*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08502

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Space Between: On Folding, Symmetries and Sampling Michal Lewandowski †, Bernhard Heinzl †, Raphael Pisoni †, Bernhard A.Moser †* Abstract Recent findings suggest that consecutive layers of neural networks with the ReLU activation function fold the input space during the learning process. While many works hint at this phenomenon, an approach to quantify the folding was only recently proposed by means of a space folding measure based on Hamming distance in the ReLU activation space. We generalize this measure to a wider class of activation functions through introduction of equivalence classes of input data, analyse its mathematical and computational properties and come up with an efficient sampling strategy for its implementation. Moreover, it has been observed that space folding values increase with network depth when the generalization error is low, but decrease when the error increases. This underpins that learned symmetries in the data manifold (e.g., invariance under reflection) become visible in terms of space folds, contributing to the network’s generalization capacity. Inspired by these findings, we outline a novel regularization scheme that encourages the network to seek solutions characterized by higher folding values. 1 Introduction Recent works in machine learning indicate that neural networks fold the input space during training ( Montúfar et al., 2014 ; Keup & Helias, 2022 ) . This phenomenon draws inspiration from the way certain natural structures—such as proteins and amino acids—fold to encode information efficiently ( Dill et al., 2008 ; Jumper et al., 2021 ) . Building on these ideas, Lewandowski et al. (2025) proposed a range-based measure in the discrete activation space of ReLU networks to quantify how much a network folds its input space as it learns. Their analysis focuses on deviations from convexity when mapping a straight-line path in the input space to the Hamming activation space: While Euclidean distances increase monotonically along the path, the corresponding Hamming distances may decrease, signaling a folding effect (cf. Fig. 1 right and Fig. 2 ). Originally developed for ReLU networks, this approach leverages the fact that the ReLU activation function partitions the input space into disjoint linear regions ( Makhoul et al., 1989 ; Montúfar et al., 2014 ) . In our paper, we firstly show that these regions correspond to equivalence classes defined by the pre-images of either \{0\} or the strictly positive interval (0,\infty) . Extending \{0\} to (-\infty,0] provides a straightforward generalization to a broader class of activation functions that accommodate negative values, including Swish ( Ramachandran et al., 2018 ) , GELU ( Hendrycks & Gimpel, 2016 ) , and SwiGLU ( Shazeer, 2020 ) . Secondly, we focus on characterizing properties of the space folding measure \chi , which do also hold in the general case. Thirdly, since computing \chi relies on sampling from different activation regions, we introduce a non-parametric sampling algorithm that exploits the structure of the aforementioned equivalence classes, thereby reducing redundant computations. Lastly, we leverage the fact that space folding values have been observed to increase with network depth when the generalization error is low, but decrease when the error increases ( Lewandowski et al., 2025 ) . We thus hypothesize the increased folding contribute to the network’s generalization capacity, and hint at a novel regularization strategy that applies the folding measure at regular intervals (e.g., every n training epochs) to induce stronger folding in the early stages and diminish its influence later in training. Our contributions are as follows. • We generalize the space folding measure beyond the ReLU activation function. Our approach relies on the fact that the pre-image of the partition \{(-\infty,0],(0,\infty)\} divides the domain into two connected sets, f^{(-1)}((-\infty,0]) and f^{(-1)}((0,\infty)) . • We state and prove general properties of the folding measure, such as ( i ) its stability under traversing different activation regions (Proposition 4.1 ), ( ii ) the sufficient and necessary, i.e., characterizing, condition for flatness (Proposition 4.2 ), ( iii ) its sensitivity to the direction of the path (Remark 4.3 ), ( iv ) invariance of flatness to direction of path (Corollary 4.4 ). • We propose a parameter-free sampling strategy in the Hamming activation space that limits steps within the same equivalence class to reduce redundancy, and we analyze its computational complexity. • We introduce a new regularization procedure for training neural networks by penalizing low space folding values. The remainder of the paper is organized as follows. Sec. 2 details related work; Sec. 3 introduces necessary concepts and fixes notation for the rest of the paper; Sec. 4 recalls the definition of the space folding measure and then provides its detailed analysis paired with the introduction of the global folding measure; Sec. 5 introduces a sampling technique from activation paths along a 1D path which relies on the Hamming distances between samples; Sec. 6 proposes a novel regularization scheme; Sec. 7 summarizes our paper and outlines future research directions. 2 Related Work Folding. The idea of folding the (input) space has been investigated, among others, in computational geometry ( Demaine et al., 2000 ) . In context of neural networks, Montúfar et al. (2014) in Section 2.4 argued that each hidden layer in a ReLU neural network acts as a folding operator, recursively collapsing input-space regions. In Phuong & Lampert (2020) , in the Appendix A.2 the authors defined the folds by ReLU networks, but left the exploration quite early on. Lewandowski et al. (2025) proposed the first measure to quantify the folding by ReLU neural networks. Our approach builds on the proposition therein, and is further motivated by the observation that folding gives rise to symmetries as discussed below. Symmetries. The modern study of symmetries (in physics) was initiated by Noether (1916) , who linked them to conservation laws: energy to time translation, momentum to space translation, and angular momentum to rotational symmetry. In the context of machine learning, researchers working with object recognition emphasised the importance of learning representations that are invariant to transformations, e.g., ( Krizhevsky et al., 2012 ) . Somewhat implicitly, symmetries have been at the core of some of the most successful deep neural network architectures, e.g., CNNs ( Fukushima, 1980 ; LeCun et al., 1989 ) are equivariant to translation invariance characteristic of image classification tasks, while GNNs ( Battaglia et al., 2018 ) are equivariant to the full group of permutations (see Higgins et al. (2022) for a detailed overview). Our work analyzes symmetries (reflection groups) that arise by space folding and their impact on the generalization capacity of the model. Linear Regions Sampling. Analyzing neural network linear regions is challenging. Early work bounded their number as a measure of expressivity in ReLU MLPs ( Montúfar et al., 2014 ; Raghu et al., 2017 ; Serra et al., 2018 ; Montúfar et al., 2021 ) , later extending to CNNs ( Xiong et al., 2020 ) and GNNs ( Chen et al., 2022 ) . Empirical studies indicate that linear regions are denser near training data ( Zhang & Wu, 2020 ) , yet standard sampling methods (e.g., Monte Carlo or Sobol sequences ( Sobol, 1967 ) ) often miss small regions. Goujon et al. (2024) showed that along one-dimensional paths, nonlinearity points scale linearly with depth, width, and activation complexity,

Read the complete text