Low-Rank Matrix Approximation for Neural Network Compression * Equal contribution as first authors. 1 Kalyan Cherukuri and Aarav Lala are with the Illinois Mathematics and Science Academy ( kcherukuri, alala1 )@imsa.edu.
Kalyan Cherukuri 1* , Aarav Lala 1* · 2025
Read Low-Rank Matrix Approximation for Neural Network Compression * Equal contribution as first authors. 1 Kalyan Cherukuri and Aarav Lala are with the Illinois Mathematics and Science Academy ( kcherukuri, alala1 )@imsa.edu. by Kalyan Cherukuri 1* , Aarav Lala 1* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20078.txt.
Introduction
Kalyan Cherukuri and Aarav Lala present a smarter way to compress neural networks, in this 2025 paper. Big networks are heavy to store and slow to run. A classic trick replaces weight tables with low-rank versions, but picking one rank for every layer wastes capacity in some and starves others.
Their method reads the spread of each layer's singular values, a measure of how much real information it holds. Layers with rich content keep higher ranks, while redundant ones shrink hard. The result is layer-wise compression with better performance and lower space and time costs than fixed-rank cuts. The pair frame rank choice as an information budget problem.
Catalog record
| CREATOR | Kalyan Cherukuri 1* , Aarav Lala 1* |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20078
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Low-Rank Matrix Approximation for Neural Network Compression * Equal contribution as first authors. 1 Kalyan Cherukuri and Aarav Lala are with the Illinois Mathematics and Science Academy ( kcherukuri, alala1 )@imsa.edu. Kalyan Cherukuri 1* , Aarav Lala 1* Abstract Deep Neural Networks (DNNs) have encountered an emerging deployment challenge due to large and expensive memory and computation requirements. In this paper, we present a new Adaptive-Rank Singular Value Decomposition (ARSVD) method that approximates the optimal rank for compressing weight matrices in neural networks using spectral entropy. Unlike conventional SVD-based methods that apply a fixed-rank truncation across all layers, ARSVD uses an adaptive selection of the rank per layer through the entropy distribution of its singular values. This approach ensures that each layer will retain a certain amount of its informational content, thereby reducing redundancy. Our method enables efficient, layer-wise compression, yielding improved performance with reduced space and time complexity compared to static-rank reduction techniques. I Introduction Deep Neural Networks (DNNs) have demonstrated success across a wide range of tasks, from computer vision to natural language processing tasks [ 1 ] . However, the high memory and computational demands of these deep models render them impractical for deployment in resource-constrained environments. The large number of hyperparameters in DNNs often exceeds the millions, resulting in high storage requirements and large computational costs, limiting both scalability and real-world applicability [ 2 ] . To address these emerging challenges, model compression has become a pressing need and an essential area for future research [ 3 ] . A promising approach is to reduce the size of neural networks while attempting to preserve performance, with a low-rank matrix approximation. Through approximating weight matrices in DNNs with lower-rank decompositions, such as with Singular Value Decomposition commonly referred to as SVD, the number of parameters involved for the network’s computation can be significantly reduced, leading to lower memory usage and faster inference times [ 3 ] . However, conventional methods that apply a fixed rank reduction towards all layers of a network may encounter suboptimal trade-offs with compression and accuracy. In this paper, we introduce a novel Adaptive-Rank Singular Value Decomposition (ARSVD) method that dynamically adapts the rank of weight matrices for each layer. This method that we propose and validate experimentally provides the foundation for a novel approach to the classical neural network compression task. To evaluate the effectiveness of our proposed algorithm, we compare the performance of several pre-trained models ResNet18, VGG16, and InceptionV3 under standard fixed-rank SVD compression, and compare results to our proposed entropy-guided ARSVD compression. These models are then tested on benchmark datasets including MNIST, CIFAR-10, and CIFAR-100. Experimental results demonstrate that ARSVD consistently achieves better compression-performance trade-offs than traditional fixed-rank SVD, preserving accuracy while significantly reducing model complexity. The adaptability of ARSVD paves the way for more computationally efficient but complex neural network architectures, particularly to expand its applicability to real-world scenarios. In this paper, we aim to contribute: an entropy-guided adaptive rank selection, performance-efficient compression, and a scalable application across different architectures. II Related Works II-A Neural Networks Neural networks have been a cornerstone of modern deep learning. Early models such as the perceptron [ 4 ] could only solve linearly separable problems, but the introduction of the backpropagation algorithm [ 5 ] enabled efficient training of multi-layer perceptrons, significantly improving their expressive power. The advent of Convolutional Neural Networks (CNNs), particularly LeNet-5 [ 6 ] , introduced architectural innovations such as local receptive fields, weight sharing, and spatial sampling concepts that became standard in deep learning. A major breakthrough came with AlexNet [ 7 ] , which outperformed traditional methods on the ImageNet benchmark and demonstrated the feasibility of training deep networks at scale. It also introduced the use of rectified linear units (ReLU) to enable robust non-linearity in deep networks. More recently, the Transformer architecture introduced in the landmark Attention Is All You Need [ 8 ] has shifted the paradigm away from convolutional and recurrent models by using self-attention mechanisms to capture global dependencies. This innovation has led to state-of-the-art performance across natural language and vision tasks. Vision Transformer (ViT) [ 9 ] further adapts this architecture to image classification, showcasing the flexibility and scalability of attention-based models in visual based tasks. II-B Space-Time Complexity Benchmarks TABLE I: Comparison of SVD-Based Compression Methods Method Complexity Performance Limitations Truncated SVD (TSVD) Time: O(kmn) Space: O(k(m+n)) Accuracy: 5–15% drop Fixed-rank, ignores layer sensitivity Randomized SVD (RSVD) Time: O(mn\log k+k^{2}(m+n)) Space: O(k(m+n)) Accuracy: 3–10% drop Requires oversampling tuning Transpose Trick Time: O(m^{3}) when m\ll n Space: No reduction Accuracy: None Only applicable when m\ll n \ell_{p} -Norm Approx. Time: Higher than \ell_{2} Space: O(k(m+n)) Accuracy: 0.63–1% drop Computationally intensive ARSVD (Proposed Method) Time: O(k(m+n)) Space: O(k(m+n)) Accuracy: 1% increase Generalization of large weight matrices SVDs have been used to shrink the computation and memory requirements of neural networks. However, most methods suffer from a loss of accuracy. A common method is the Truncated SVD (TSVD) for Low Rank Approximation. Here we will present the top existing methods in the literature: Table I compares SVD-based compression methods in neural networks. • TSVD [ 10 ] and RSVD [ 11 ] offer efficient inference but with accuracy trade-offs • Transpose trick [ 12 ] is exact but situationally useful • \ell_{p} -norm [ 13 ] preserves accuracy at higher cost Key observations: • TSVD [ 10 ] and RSVD [ 11 ] offer linear inference complexity but suffer accuracy losses • The transpose trick [ 12 ] provides exact decomposition but has limited applicability • \ell_{p} -norm methods [ 13 ] preserve accuracy better at higher computational cost II-C Singular Value Decomposition Limitations Singular Value Decomposition (SVD) is widely used and has become something of the norm for matrix factorization tasks, with significant application in dimensionality reduction and data compression. Despite its wide appeal, SVD faces several limitations that are extremely critical in the context of deep neural network compression. SVD is computationally expensive, with a time complexity of O(\min(m^{2}n,mn^{2})) , which renders it inefficient for large-scale matrices commonly encountered in deep networks. To mitigate this, techniques such as randomized SVD and truncated SVD have been introduced [ 14 ] . These methods reduce the computation by approximating or truncating singular values. However, this often results in a drop in accuracy and performance due to neglecting informative spectral components. For our compression task, preserving the meaningful structures present in the weight matrices is critical to maintaining performance. Although truncated or randomized approaches speed up computation, they ignore significant components in the matrix’s singular value entropy distribution. Heuristically preserving features of the weight matrices is therefore a core requirement in high-performance neural network compression. SVD is also sensitive to noise, often capturing variations that do not reflect the underlying data structure. Existing robust variants such as Proper Orthogonal Decomposition attempt to mitigate this through minimizing reconstruction error under noisy conditions [ 15 ] . Nevertheless, handling noise effectively remains a challenge, and continues to be an active area of matrix approximation research [ 16 ] . To address the scalability of SVD in high-dimensional contexts, fast SVD variants using a block-wise or incremental strategy have been developed [ 17 ] . These methods