Linearity-based neural network compression
Silas Dobler, Florian Lemmerich · 2025
Read Linearity-based neural network compression by Silas Dobler, Florian Lemmerich (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-21146.txt.
Introduction
Silas Dobler and Florian Lemmerich (2025) shrink neural networks by exploiting neurons that behave almost linearly. Most compression methods cut weights by importance or redundancy scores. The authors start from a different fact: with ReLU-style activations, a neuron that is nearly always on acts like a straight line. Two layers joined by such neurons can be merged into one. They develop the theory for when this merging is safe.
In tests, most models compressed to a quarter of their size with no loss of accuracy. The method also stacked well on models already pruned by importance scores, with little interference. That independence matters: the tricks find different waste. The authors present linearity-based compression as a new family of methods, not a tweak of pruning. Smaller models from it would run cheaper on phones and embedded hardware. They frame the paper as the foundation for that family.
Catalog record
| CREATOR | Silas Dobler, Florian Lemmerich |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-21146
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Linearity-based neural network compression Silas Dobler, Florian Lemmerich Abstract In neural network compression, most current methods reduce unnecessary parameters by measuring importance and redundancy. To augment already highly optimized existing solutions, we propose linearity-based compression as a novel way to reduce weights in a neural network. It is based on the intuition that with ReLU-like activation functions, neurons that are almost always activated behave linearly, allowing for merging of subsequent layers. We introduce the theory underlying this compression and evaluate our approach experimentally. Our novel method achieves a lossless compression down to 1/4 of the original model size in over the majority of tested models. Applying our method on already importance-based pruned models shows very little interference between different types of compression, demonstrating the option of successful combination of techniques. Overall, our work lays the foundation for a new type of compression method that enables smaller and ultimately more efficient neural network models. 1 Introduction The number of parameters in deep neural networks has dramatically grown in the last decade, raising fundamental concerns about the computational and thus indirectly environmental costs. Concurrently, use cases of the deep learning models also expanded to systems with restricted computational power or memory, such as mobile phones or embedded systems. [ Andrew G. Howard et al. 2017 ] To reduce costs and enable distributed usage of deep learning models, network compression attempts to reduce the size of existing models to a manageable size with minimal functional impact. [ Neill 2020 ] In this direction, several highly optimized ways to reduce model size have been proposed, including quantization, removing unimportant weights or combining redundant weights. State-of-the-art methods combine best efforts of all approaches to achieve the highest compression while maintaining as much performance as possible. [ Han et al. 2015 ] The main objective of this work is to introduce a fundamentally new approach for compressing neural networks based on linearity . Linearity of an activation function allows redistributing weights and removing the neuron itself. Linear neurons are the most active neurons and therefore not widely considered for compression in other approaches. An up-following research question addresses the compression performance of this approach in comparison with existing methods. The main idea of this work is to exploit the partial linearity of neurons with ReLU-like activations to multiply out their weights into the next layer. We start with theoretical considerations regarding this idea, which allows reshaping the model to use fewer parameters with guaranteed identical outputs under certain conditions. Next, we demonstrate how these conditions can be relaxed to develop a practical approach that enables to prune more neurons but only approximates the original output. Figure 1 summarizes our approach: Neurons with (theoretically or empirically) linear activations only are replaced with direct connections between their input and output layer. Finally, the compression performance of this prototype is experimentally evaluated on both a standard fully connected neural network and a network that has already been pruned using an alternative method. The main contribution of this work is the introduction of a novel way of pruning neural networks. For this approach, we present a theoretical foundation that provides a justification for further compression methods based on linearity. Furthermore, we demonstrate that our prototype algorithm exploiting this idea leads for many datasets to substantial compression in network size while maintaining prediction performance. The code of our implementation is available for others to adapt or build upon. 1 1 1 https://github.com/sdobler19/linearity-based-neural-network-compression Figure 1 : Example of linearity-based network compression. Replacing all linear neurons in the second layer of the original model with a shortcut connection between the neurons input and output layer. 2 Mathematical Justification The idea underlying our proposed approach is to exploit the ineffectiveness of neurons with a linear activation function in the network. We begin by demonstrating how a single linear neuron can be eliminated by appropriately redistributing its weights. Next, we explore how even with non-linear ReLU activation, certain neurons may behave linearly under specific conditions. Finally, we examine the impact of this approach on the overall size of the model. 2.1 Removal of Linear Neurons As it is well known, in a standard feedforward neural network with exclusively linear activation functions, linear neurons can be collapsed into the next layer of the network by multiplying out the parameters. As a result, the adjusted weights and biases in the following layer account for the inputs originally processed by the removed linear neurons. The linear neuron is dissolved in the process. We consider a fully connected neural network with k hidden layers, denoted h_{i} for i=1,\ldots,k-1 , where each layer h_{i} contains m_{i} neurons. Assume that all but one neuron use a ReLU activation, and we denote their outputs h_{ij} , with associated weights w^{(ij)}_{h_{lm}} and biases b_{ij} . As the exception, we let (without loss of generality) the first neuron of layer i have a linear activation function f(x) , i.e., f(x)=x . Then, the next layer’s neurons then can be represented by \displaystyle h_{i+1p}=ReLU((\sum_{j=1}^{m_{i}}w^{(i+1p)}_{h_{ij}}h_{ij})+b_{i+1p})\hskip 10.00002pt\text{for }p=1,...,m_{i+1} (1) Layer h_{i} can be represented in the same way, with adaption to the linear first neuron. \displaystyle h_{i}=((\sum_{j=1}^{m_{i-1}}w^{(i-1,1)}_{h_{i-1j}}h_{i-1j})+b_{i-1,1},ReLU(...),...,ReLU((\sum_{j=1}^{m_{i-1}}w^{(i-1,m_{1})}_{h_{i-1j}}h_{i-1j})+b_{i-1,m_{i}})) (2) Inserting 2 into 1 and exploiting the linearity gives the following representation for the neurons in the next layer: \displaystyle h_{i+1p} \displaystyle=ReLU((w^{(i+1p)}_{h_{i1}}((\sum_{l=1}^{m_{i-1}}w^{(i1)}_{h_{i-1l}}h_{i-1l})+b_{i1})+\sum_{j=2}^{m_{i}}w^{(ip)}_{h_{ij}}h_{ij})+b_{i+1p}) \displaystyle=ReLU((\sum_{j=2}^{m_{i}}w^{(ip)}_{h_{ij}}h_{ij})+(\sum_{l=1}^{m_{i-1}}w^{(i+1p)}_{h_{i1}}w^{(i1)}_{h_{i-1l}}h_{i-1l})+w^{(i+1p)}_{h_{i1}}b_{i1}+b_{i+1p}) \displaystyle=ReLU((\sum_{j=2}^{m_{i}}w^{(ip)}_{h_{ij}}h_{ij})+(\sum_{l=1}^{m_{i-1}}w^{(i1)}_{h_{i-1l}--updated}h_{i-1l})+b_{i+1p--updated}) (3) for p=1,...,m_{i+1} . The weights w^{(ij)} of a linear neuron h_{ij} are updated by multiplying them by the weight corresponding to the output of the linear neuron in each neuron of the next layer w^{(i+1p)}_{h_{i1}} before adding the resulting weights to those neurons. The same updating rule applies to the bias. To ensure the network produces identical predictions, the inputs originally received by the linear neuron must also be provided directly to the neurons in the next layer. The linear neuron and all its corresponding weights can then be removed. Now that we know that a linear neuron can be removed, we will see how it can happen that neurons with the non-linear activation of ReLUs still end up being linear. 2.2 Linearity of ReLUs ReLU activation functions operate by outputting zero for negative inputs and applying the identity function f(x)=x for non-negative inputs. The special setting in the neural network can lead to conditions where all possible inputs to a ReLU are non-negative. Such a ReLU with restricted domain ReLU:\mathbb{R}^{+}_{0}\rightarrow\mathbb{R}^{+}_{0} allows the conclusion, ReLU(x)=max(0,x)=x (4) resulting in linear ReLUs, which could be removed as previously seen. For a ReLU input in the neural network (f(x) = \sum_{i=1}^{n}x_{i}w_{i}+b ) to always be non-negative, two conditions must hold: all neuron inputs x_{i} must be non-negative, and all weights w_{i} and the bias must be non-negative. The first is guaranteed in deeper layers if all hidden layers use non-negative activation functions like ReLU. The second can be verified directly in a trained or pre-trained model. Thus, in a feedforward network with ReLU activations, any neuron from the second layer onward with non-negative weights and bias behaves linearly and can be removed. 2.3 Combination of Multiple Linear Neurons The number of parameters associated with a single neuron in a fully connected network depends on the number of neurons in the previous layer width_{i-1} and the number of neurons in the next layer width_{i+1} . Without considering any