Complexity of Linear Regions in Self-supervised Deep ReLU Networks
Mufhumudzi Muthivhi, Terence L. van Zyl · 2026
Read Complexity of Linear Regions in Self-supervised Deep ReLU Networks by Mufhumudzi Muthivhi, Terence L. van Zyl (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24393.txt.
Introduction
Muthivhi and van Zyl track how linear regions evolve in self-taught deep ReLU networks. Taught, contrastive, and self-taught methods are tested on MNIST and FashionMNIST, using SplineCam to pull flat shapes. Region counts, areas, shapes, and edges are tracked, and region change tracks with model quality. Self-taught methods build far fewer regions for accuracy like that of taught models. Contrastive methods grow regions fast, while self-taught methods merge nearby regions. Region shape reveals model collapse early, suggesting shape scores show model quality and performance.
Catalog record
| CREATOR | Mufhumudzi Muthivhi, Terence L. van Zyl |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27825 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.24393
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Complexity of Linear Regions in Self-supervised Deep ReLU Networks Mufhumudzi Muthivhi, Terence L. van Zyl Abstract There has been growing interest in studying the complexity of Rectified Linear Unit (ReLU) based activation networks. Recent work investigates the evolution of the number of piecewise-linear partitions (linear regions) that are formed during training. However, current research is limited to examining the complexity of models trained in a supervised way. Self-Supervised Learning (SSL) differs in that it directly optimises the representation space using a loss function to enhance the model’s performance across multiple downstream tasks. This study investigates the local distribution of linear regions produced by SSL models. We demonstrate that the evolution of linear regions correlates with the representation quality by utilising SplineCam to extract two-dimensional polytopes near the data distribution. We track the number, area, eccentricity, and boundaries of regions throughout training. The study compares supervised, contrastive, and self-distillation methods over two standard benchmark datasets, MNIST and FashionMNIST. The analysis of the experimental results shows that self-supervised methods create substantially fewer regions to achieve comparable accuracy to supervised models. Contrastive methods rapidly expand regions over time, whereas self-distillation methods tend to consolidate by merging neighbouring regions. Lastly, we can detect representation collapse early within the geometric space of linear regions. Our analysis suggests that polytopal metrics can serve as reliable indicators of representation quality and model performance. 1 Introduction Figure 1 : Self-supervised methods (SimCLR, BYOL, SimSiam) produce fewer linear regions than their supervised counterparts (Classifier, Triplet, SupCon) while achieving comparable accuracy. Both contrastive and self-distillation approaches leverage positive pairings, resulting in the rapid merging of linear regions due to shared activation patterns. The opacity reflects the progression of training over epochs. Low-opacity points depict results achieved early in training, while darker points represent results achieved later. Continuous Piecewise Linear (CPWL) neural networks partition the input space into a finite set of convex polytopes, referred to as linear regions 15 . Within each region, the network applies a single affine transformation. Regions are formed by applying a nonlinear activation function over the output values of the affine transformation. The set of all linear regions produced by a neural network defines a piecewise-linear tessellation of the input space. Understanding the structure and distribution of regions near the data distribution has become an active area of research. Practitioners count the number of regions as a measure of network complexity 27 ; 33 ; 34 . 20 propose SplineCam for interpretation and visualisation of the exact number of regions from a two-dimensional input space 20 . Most current work focuses on analysing the linear regions generated by supervised classifier networks 15 ; 22 . Some studies have explored the local geometry of generative models 21 ; 19 . More recent work examines the complexity of input space partitioning in spiking neural networks 29 . However, the geometric properties of Self-Supervised Learning (SSL) representations remain largely underexplored. SSL has shown remarkable performance on several downstream tasks 9 ; 2 ; 35 . It intrinsically generates supervision signals by augmenting input data. This work investigates the local distribution of linear regions produced by SSL methods. We transform a high-dimensional input space into a two-dimensional space and partition it to obtain its linear regions. Specifically, we consider contrastive and self-distillation methods 5 ; 17 ; 6 ; 13 . Self-supervised contrastive learning captures the similarity among instances. They minimise the distance between positive pairs within the embedding space while simultaneously trying to keep negative pairs apart. However, they require large batch sizes or memory banks to store sufficient negative samples, thereby preventing overfitting. Self-distillation methods forgo the need for negatives by using a predictor network to predict the representation of views from the same image. Several specific conditions have been proposed to prevent representations from collapsing into degenerate solutions 36 ; 23 ; 40 . We hypothesise that different SSL objectives produce unique geometric properties in the partitioned input space. In contrastive methods, positive pairs encourage samples to share activation patterns, essentially merging regions together. Negative pairs exert a repulsive force that tears regions apart. In contrast, self-distillation methods do not use negative pairs; thus, regions merge rapidly. This effect reflects on the heightened risk of representation collapse for self-distillation methods. Figure 1 contrasts the performance of supervised and self-supervised methods. All the SSL methods produce fewer regions and achieve accuracy comparable to that of supervised variants in terms of performance. Furthermore, results show that the contrastive objective produces a larger number of isotropic small regions that fit the in-distribution data well. Self-distillation encourages fewer, larger, and more anisotropic partitions, which correlates with improved generalisation performance. We contribute to the existing literature by: • track the number, area, eccentricity, and boundaries of regions throughout training from a projected high-dimensional input space, • compare the evolution of regions from supervised, contrastive and self-distillation methods, • simulates the effects of representation collapse within the polytopal space, • provides a qualitative analysis of linear regions and their density within the projected high-dimensional input space. 2 Related Work 2.1 Continuous Piecewise-Linear Functions A neural network consists of one or more linear transformations followed by a non-linear operation. Current deep networks predominantly use a Continuous Piecewise-Linear (CPWL) nonlinearity such as a Rectified Linear Unit (ReLU), leaky-ReLU or MaxOut operation 12 . CPWL functions allow practitioners to assess the complexity of a neural network by counting the number of linear partitions (linear regions) formed by a network. 27 show that the number of linear regions grows exponentially with depth 27 . 32 relate the number of regions to expressivity and generalization 32 . 30 demonstrate that neural networks have strong robustness near the training data distribution, especially with data augmentation and ReLU activations 30 . 34 provides tighter bounds on the number of regions compared to previous studies 34 . Several works utilise CPWL functions to investigate properties in generative models, Recurrent Neural Networks, or spiking neural networks, including the understanding of classifiers, out-of-distribution detection, and the interpretation and visualisation of regions emerging from supervised classifiers 21 ; 22 ; 4 ; 10 ; 24 ; 29 ; 20 ; 15 . 2.2 Self-supervised Learning Recently, Self-supervised Learning (SSL) methods have achieved performance comparable to that of their supervised counterparts across a range of downstream tasks 35 . SSL utilises a formulated pretext-task to generate several pseudo targets from the input data 14 . An objective function learns a representation space that is invariant to augmentations made to the input data. Early work utilised contrastive learning to associate positive pairs with each other while pushing negative pairs apart 5 ; 17 . Self-distillation methods maximise the similarity between two augmented views of the same image 6 ; 13 . One limitation is that they do not explicitly include a repulsive force (negative examples) to prevent the representations from collapsing into degenerate solutions. Hence, the majority of work in this area focuses on devising implicit architectural constraints and optimisation strategies, such as a prediction head, stop-gradient operation, or a redundancy-reduction objective 39 ; 3 ; 2 . Figure 2 : Illustration of how a ReLU network partitions the input space into linear regions. The orange neurons are active. The distinct activation pattern corresponds to