Concatenated Matrix SVD: Compression Bounds, Incremental Approximation, and Error-Constrained Clustering
Maksym Shamrai [email protected] · 2026
Read Concatenated Matrix SVD: Compression Bounds, Incremental Approximation, and Error-Constrained Clustering by Maksym Shamrai [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11626.txt.
Introduction
Maksym Shamrai studies shrinking large sets of matrices by SVD. The 2026 preprint covers joined-matrix SVD in depth. Big matrix sets arise across machine learning and computing. Signal work and science computing add their own sets.
The standard trick concatenates the matrices into one. A truncated SVD then compresses the joined matrix. That enables parameter sharing and efficient reconstruction. Shamrai derives size bounds for the strategy. A step-wise method updates the shrinkage as data arrive. Error-constrained clustering groups matrices under a set error cap. The results serve multi-view learning and network compression. Users gain theory for a widely adopted practice.
Catalog record
| CREATOR | Maksym Shamrai [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11626
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Concatenated Matrix SVD: Compression Bounds, Incremental Approximation, and Error-Constrained Clustering Maksym Shamrai [email protected] Abstract Large collections of matrices arise throughout modern machine learning, signal processing, and scientific computing, where they are commonly compressed by concatenation followed by truncated singular value decomposition (SVD). This strategy enables parameter sharing and efficient reconstruction and has been widely adopted across domains ranging from multi-view learning and signal processing to neural network compression. However, it leaves a fundamental question unanswered: which matrices can be safely concatenated and compressed together under explicit reconstruction error constraints? Existing approaches rely on heuristic or architecture-specific grouping and provide no principled guarantees on the resulting SVD approximation error. In the present work, we introduce a theory-driven framework for compression-aware clustering of matrices under SVD compression constraints. Our analysis establishes new spectral bounds for horizontally concatenated matrices, deriving global upper bounds on the optimal rank- r r SVD reconstruction error from lower bounds on singular value growth. The first bound follows from Weyl-type monotonicity under blockwise extensions, while the second leverages singular values of incremental residuals to yield tighter, per-block guarantees. We further develop an efficient approximate estimator based on incremental truncated SVD that tracks dominant singular values without forming the full concatenated matrix. Therefore, we propose three clustering algorithms that merge matrices only when their predicted joint SVD compression error remains below a user-specified threshold. The algorithms span a trade-off between speed, provable accuracy, and scalability, enabling compression-aware clustering with explicit error control. Code is available online https://github.com/mshamrai/concatenated-matrix-svd . 1 Introduction Large collections of matrices naturally arise in a wide range of applications, including multi-view representation learning, temporal aggregation of features, scientific simulations, neural network activations, and model parameter compression. A common and effective strategy for compressing such data is to horizontally concatenate matrices (or matrix-shaped tensors) and apply a truncated Singular Value Decomposition (SVD) to the resulting large matrix ( 1 ; 25 ; 23 ; 13 ) . This approach yields a single low-rank factorization that enables parameter sharing, compact representation, and direct reconstruction of the original blocks. Compared to tensor decomposition methods such as Tucker or higher-order SVD (HOSVD) ( 28 ; 6 ; 7 ; 14 ) , matrix-based concatenation followed by SVD offers several practical advantages. It produces a single latent basis shared across all blocks, avoids multilinear contractions at reconstruction time, and integrates naturally with pipelines that operate on matrix-valued embeddings or linear transformations. As a result, concatenated SVD has become a standard tool in domains ranging from signal processing to modern large-scale representation learning. Despite its widespread use, a fundamental problem remains unresolved. Given a large collection of matrices, how can we efficiently determine which subsets should be concatenated and compressed together so that their joint low-rank approximation error remains below a prescribed tolerance? In practice, existing compression pipelines rely on heuristic grouping strategies, architectural constraints, or domain-specific assumptions to decide which matrices share a low-rank basis ( 30 ; 17 ; 16 ) . Such approaches provide no explicit control over the resulting SVD reconstruction error and offer no guarantees that merging additional matrices will not violate accuracy requirements. At a high level, the difficulty stems from the fact that concatenating matrices fundamentally alters the spectrum of the resulting matrix. While individual blocks may admit accurate low-rank approximations, their concatenation can increase effective rank and degrade approximation quality. Determining whether two or more matrices can safely share a low-rank representation therefore requires understanding how the singular values of the concatenated matrix evolve as new blocks are appended. This is inherently a spectral question and cannot be addressed by distance-based clustering or purely geometric similarity measures, as used in classical clustering methods. In this work, we introduce the first theoretically grounded and computationally efficient framework for compression-aware clustering of matrices under explicit SVD reconstruction constraints. Our approach is based on new spectral bounds that characterize how singular values evolve under horizontal concatenation. Specifically, we derive two lower bounds on the singular values of concatenated matrices using classical matrix perturbation theory. The first bound follows from Weyl-type monotonicity under blockwise extensions ( 32 ; 24 ) , providing conservative but fast guarantees. The second, sharper bound exploits the singular values of incremental residuals, capturing the orthogonal contribution of each appended block and yielding tighter per-block estimates. Leveraging these spectral lower bounds, we derive corresponding global upper bounds on the optimal rank- r SVD reconstruction error of the concatenated matrix. These bounds are exact in the non-truncated case and remain tight in practice under truncation. To enable scalability, we further develop an efficient approximate estimator based on incremental truncated SVD. This estimator maintains approximate dominant singular values as blocks are appended, drawing on ideas from incremental PCA and SVD ( 12 ; 15 ; 2 ) and randomized low-rank approximation ( 11 ; 27 ) , while avoiding construction of the full concatenated matrix. Importantly, we do not claim a new incremental SVD algorithm, rather, we show how incremental singular value estimation can be repurposed as a decision-making primitive for compression-aware clustering under explicit error budgets. Building on these theoretical results, we design three clustering algorithms that explicitly enforce SVD compression constraints. The first is a fast, instantaneous method based on Weyl-monotone bounds. The second provides provable reconstruction accuracy using residual-based spectral bounds. The third is a scalable approximate method that employs incremental truncated SVD to balance efficiency and approximation quality. All three algorithms merge matrices only when the predicted SVD reconstruction error of their concatenation lies below a user-specified threshold, thereby providing explicit and interpretable accuracy control. In summary, this work establishes concatenation-aware SVD compression as a principled foundation for clustering and compressing large collections of matrices. By unifying spectral perturbation theory, incremental singular value estimation, and compression-driven clustering, we provide both theoretical guarantees and practical algorithms. 2 Related Work SVD-based compression of matrix collections. Applying truncated singular value decomposition (SVD) to a matrix obtained by horizontally concatenating multiple blocks is a widely used technique in signal processing, computer vision, scientific computing, and representation learning. This strategy provides a simple and effective compression mechanism based on a single low-rank factorization, enabling direct reconstruction of the original matrices without higher-order tensor contractions or complex decoding procedures ( 13 ; 11 ) . Concatenated SVD has been successfully applied across a broad range of domains. In large language models, joint SVD of concatenated weight matrices has been used to share low-rank projections across attention components, layers, or experts, enabling parameter reduction while preserving accuracy. Representative examples include unified QKV decompositions ( 30 ) , intra-layer shared projections ( 17 ) , cross-layer parameter sharing ( 29 ) , and expert merging in mixture-of-experts architectures ( 16 ; 4 ) . In these settings, concatenation is typically guided by architectural structure (e.g., matrices belonging to the same layer or module) or semantic similarity (e.g., adjacent layers or related experts). Related ideas also appear in wireless signal processing, where concatenated SVD is used to design shared precoders across frequency bands ( 34 ) , as well as in neuroscience and genomics, where large collections of measurements are concatenated to obtain global low-dimensional representations across sessions, experimental conditions, or chromosomes ( 21 ; 35 ) . Across these application