Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks

Ying Chen, Aoxi Li, Jihun Kim, Javad Lavaei · 2026

Read Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks by Ying Chen, Aoxi Li, Jihun Kim, Javad Lavaei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05659.txt.

Introduction

Lean networks replace dense layers with low-rank ones, but their power limits are poorly mapped. Ying Chen, Aoxi Li, Jihun Kim, and Javad Lavaei study networks built from diagonal plus low-rank parts. They first expose a paradox in purely rank-1 layers: such layers can fit arbitrary scalar data exactly, yet collapse when asked to approximate functions.

Adding a diagonal component breaks the bottleneck without giving up the savings. The authors develop a structural match and broad approximation results for the combined design. Their analysis avoids strict assumptions like pretrained base matrices or specific activations.

Catalog record

CREATORYing Chen, Aoxi Li, Jihun Kim, Javad Lavaei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05659

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks Ying Chen, Aoxi Li, Jihun Kim, Javad Lavaei Abstract The massive computational costs of scaling modern deep learning architectures have driven the widespread use of parameter-efficient low-rank structures, such as LoRA and low-rank factorization. However, theoretical guarantees for their expressive power are less explored, often relying on restrictive priors like a pretrained base matrix, ReLU activations or non-verifiable singularity conditions. We first investigate the limits of neural networks constrained strictly to low-rank manifolds without pretrained dense priors. We demonstrate a theoretical paradox: while purely rank-1 layers can exactly interpolate arbitrary scalar datasets, they collapse for function approximations. To overcome this bottleneck without surrendering parameter efficiency, we introduce a unified Structural Correspondence framework. We prove that augmenting low-rank layers with only a minimal sparse diagonal component, say a Diagonal plus Low-Rank (DLoR) structure, is sufficient to reach Universal Approximation. We show that any full-rank transformation can be exactly reconstructed using these DLoR components by trading off network width (additive decomposition) or depth (multiplicative decomposition). By tracking asymptotic Taylor remainders, we prove that DLoR neural networks fully restore the Universal Approximation Theorem for general activation functions. Finally, we establish that multiplicative depth provides superior parameter-to-expressivity scaling compared to additive width. Our results show that dense matrices and specific activation functions are not topological prerequisites for universal expressivity. 1 Introduction (a) Interpolation by rank-1 NN (b) Additive and multiplicative decomposition Figure 1 : Overview of our work. (a) Rank-1 neural networks (NN) can perfectly interpolate any set of scalar outputs, but inherently fail for multi-dimensional function approximations. (b) We use rank-deficient structures by either expanding width (with additive decomposition) or extending depth (with multiplicative decomposition). While the former can rely on purely low-rank weight matrices, the latter necessitates DLoR weight matrices. The era of modern foundation models has been defined by the aggressive scaling of deep neural network architectures to billions, or even trillions of parameters ( lecun2015deep ; openai2023gpt4 ) . While highly over-parameterized neural networks are appealing due to their favorable optimization landscapes and global convergence properties ( allenzhu2019convergence ; baldi2024shallow ) , their immense scale imposes prohibitive memory and computational bottlenecks ( rajbhandari2020zero ; strubell2019energy ) . This reality has driven a critical need for parameter and memory-efficient training frameworks ( lialin2023scaling ) , as well as robust network compression techniques to alleviate the burden of deployment. Techniques such as Low-Rank Adaptation (LoRA) ( hu2022lora ; lialin2023relora ) , sparse plus low-rank matrix factorization ( chandrasekaran2009sparse ; ding2023sparse ) , and parameter-efficient adapter modules ( houlsby2019parameter ) have become the preferred standard for efficiency. These low-rank structures are highly favored because they drastically reduce computational overhead while leveraging the phenomenon that pre-trained neural networks typically operate within a low intrinsic dimensionality ( aghajanyan2021intrinsic ) . By operating on constrained low-rank manifolds, these methods can yield empirical fine-tuning performance that rivals full-rank training ( zhang2026sparsityawarelowrankrepresentationefficient ) . However, purely low-rank structures restrict parameters to a severely constrained low-dimensional subspace. Specifically, low-rank components are restricted to capturing only the dominant singular value spectrum, entirely ignoring variations outside this subspace. This prompts a question: what exact topological or geometric bottlenecks constrain the expressive power of severely rank-deficient neural networks? Recent literature has attempted to justify the expressive power of parameter-efficient models ( zeng2023expressive ; giannou2023expressive ; li2025approximation ) . However, the theoretical guarantees provided by these works remain limited. Current claims rely on the presence of a pretrained dense base matrix, the specific use of the ReLU or Swish activation function, or non-verifiable singularity conditions regarding the weight matrices. In contrast, we ask a more fundamental question: can a strictly constrained neural network achieve the Universal Approximation Theorem (UAT) ( hornik1989multilayer ; kidger2020universal ) , without these restrictive priors and assumptions? We first investigate this by formalizing the interpolation limits of purely low-rank architectures under general activation functions. We demonstrate a paradox: while a single purely rank-1 hidden layer exhibits a misleading exact interpolation capability for scalar datasets, this capacity fundamentally collapses when extended to multi-dimensional function approximation. We show that strict rank-1 mappings suffer from a geometric phenomenon that we term orthogonal blindness . Because the network evaluates inputs strictly through a 1-dimensional projection, it becomes completely invariant to variations in the data orthogonal to this projection vector, rendering it mathematically incapable of spanning higher-dimensional spaces. Motivated by recent empirical frameworks like SLTrain ( han2024sltrain ) , which demonstrate that combining low-rank factors with randomly supported sparse matrices successfully recovers full-rank performance, we hypothesize that the expressive power of low-rank neural networks can be fully restored by a limited sparse augmentation. We consider a matrix structure of the form M=\alpha I+UV^{T} , a special instance of a sparse plus low-rank matrix that we term a Diagonal plus Low-Rank (DLoR) component. It introduces a minimal sparse diagonal component to the low-rank structure \operatorname{rank}(UV^{T})\leq r , where r is any arbitrary rank constraint. Unlike a dense prior, this \alpha I component requires only one additional parameter to store. Yet, it provides the vital intrinsic residual pathway required to capture the orthogonal spectrum, allowing information to bypass the low-dimensional subspace and breaking the geometric bottleneck of orthogonal blindness. Crucially, we establish a Structural Correspondence framework for decomposing any full-rank transformation using DLoR components. We prove that restructuring a weight matrix via an additive decomposition naturally translates to expanding the network’s width, mapping a high-rank linear operation onto a parallel summation of nonlinear layers. Conversely, we prove that a multiplicative decomposition corresponds to extending the network’s depth. This structural correspondence provides a framework for post-training or fine-tuning large models, allowing the parameter space to be effectively decoupled and optimized in parallel. A summary of our theoretical framework, contrasting the interpolation failure of purely rank-1 neural networks with our DLoR architectures, is provided in Figure 1 . Specifically, our main contributions are as follows: • Orthogonal Blindness of Strict Low-Rank Neural Networks: We formalize the expressive limits of purely rank-1 neural networks. We prove that while they can interpolate any scalar targets, they fundamentally suffer from orthogonal blindness for general continuous function approximation. Thus, purely low-rank representations are insufficient for function approximation. • The Structural Correspondence Framework: We introduce both additive and multiplicative decomposition methods using DLoR components. We prove that rank deficiency can be compensated for by either expanding network width (additive) or extending network depth (multiplicative). • Universal Approximation via DLoR Structures : We prove that introducing only a minimal \mathcal{O}(1) sparse diagonal to form a DLoR structure perfectly overcomes orthogonal blindness. We show that Deep DLoR Neural Networks with bounded width restore uniform approximation capabilities for continuous functions with any nonaffine continuous activation functions. 2 Related Work Universal Approximation and the Depth-Width Trade-off. The Universal Approximation Theorem (UAT) establishes that neural networks are dense in the space of continuous functions on a compact domain, C(\mathcal{K},\mathbb{R}^{m}) , with respect to the uniform norm. Classic results proved that a single hidden layer utilizing any non-polynomial activation function can achieve uniform \epsilon -approximation ( hornik1989multilayer ; cybenko1989approximation ; pinkus1999approximation ) . However, these shallow architectures suffer from the curse of dimensionality: to approximate functions in a bounded domain \mathcal{K}\subset\mathbb{R}^{n} , shallow architectures require a prohibitive width

Read the complete text