CR-Net : Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure

Boao Kong [email protected], Junzhu Liang [email protected], Yuxi Liu [email protected], Renjia Deng [email protected], Kun Yuan [email protected] · 2025

Read CR-Net : Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure by Boao Kong [email protected], Junzhu Liang [email protected], Yuxi Liu [email protected], Renjia Deng [email protected], Kun Yuan [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18993.txt.

Introduction

Training very large language models costs huge memory and computing power, and low-rank designs try to cut both. Boao Kong and four co-authors start from a finding about what flows between the layers of such a model. The changes in layer activations, they found, are themselves low-rank, meaning a small summary captures most of them. Past low-rank methods tend to lose accuracy, add overhead, or save little memory.

Their design, CR-Net, rebuilds each layer's activations from the prior layer's output plus its low-rank change. A tailored recomputation scheme cuts the memory needed to store activations during training. In training runs from small models up to seven billion parameters, CR-Net beat other low-rank methods. It did so while using fewer resources and less memory.

Catalog record

CREATORBoao Kong [email protected], Junzhu Liang [email protected], Yuxi Liu [email protected], Renjia Deng [email protected], Kun Yuan [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-18993

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CR-Net : Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure Boao Kong [email protected], Junzhu Liang [email protected], Yuxi Liu [email protected], Renjia Deng [email protected], Kun Yuan [email protected] Abstract Low-rank architectures have become increasingly important for efficient large language model (LLM) pre-training, providing substantial reductions in both parameter complexity and memory/computational demands. Despite these advantages, current low-rank methods face three critical shortcomings: (1) compromised model performance, (2) considerable computational overhead, and (3) limited activation memory savings. To address these limitations, we propose C ross-layer Low- R ank residual Net work ( CR-Net ), an innovative parameter-efficient framework inspired by our discovery that inter-layer activation residuals possess low-rank properties. CR-Net implements this insight through a dual-path architecture that efficiently reconstructs layer activations by combining previous-layer outputs with their low-rank differences, thereby maintaining high-rank information with minimal parameters. We further develop a specialized activation recomputation strategy tailored for CR-Net that dramatically reduces memory requirements. Extensive pre-training experiments across model scales from 60M to 7B parameters demonstrate that CR-Net consistently outperforms state-of-the-art low-rank frameworks while requiring fewer computational resources and less memory. ${}^{1}$ ${}^{1}$ footnotetext: Peking University. ${}^{*}$ ${}^{*}$ footnotetext: Equal contribution. ${}^{\dagger}$ ${}^{\dagger}$ footnotetext: Corresponding author. Kun Yuan is also affiliated with National Engineering Labratory for Big Data Analytics and Applications, and AI for Science Institute, Beijing, China. 1 Introduction Large language models (LLMs) have achieved remarkable success across diverse domains [ 6 , 58 , 18 , 39 ] , with strong empirical evidence demonstrating that scaling both model parameters and training data consistently improves model performance [ 33 , 24 , 51 ] . However, as model sizes grow from millions to billions of parameters, the computational requirements for pre-training increase exponentially in both computation power and memory consumption. This necessitates months-long training cycles on large clusters of high-performance GPUs, making the process both time-prohibitive and economically challenging. For example, scaling from GPT-3’s 2.7B parameter version to its 175B counterpart increases memory requirements from 21GB to 1.4TB (a 66× increase) and computational costs from 55 to 3,640 petaflop-days (another 66× increase) [ 6 ] . These challenges underscore the critical need for developing efficient architectures that systematically leverage the inherent characteristics of LLM model structures to optimize both memory utilization and computational efficiency while maintaining cost-effectiveness. Low-rank structures in LLM training. The low-rank property has emerged as one of the most prominent structural characteristics in transformer-based models, attracting significant research attention due to its consistent empirical validation. Existing approaches to leveraging this property can be fundamentally classified based on two principal observations: (O1). Low-rank parameter. Representative approaches such as Low-Rank Adaptation (LoRA) and its variants [ 26 , 37 , 63 , 42 , 60 , 32 ] utilize learnable low-rank parameter matrices as memory-efficient substitutes for full-rank weight updates. This design achieves significant parameter reduction while preserving model capacity. Furthermore, this paradigm demonstrates strong compatibility with sparsity-inducing techniques (e.g., SLTrain [ 19 ] and LOST [ 36 ] ) to achieve improved model performance under constrained parameter budgets. (O2). Low-rank gradient. Recent research has explored the intrinsic low-rank properties of LLM gradients [ 64 , 8 , 21 , 27 , 53 , 23 , 38 ] . By projecting optimizer states into low-dimensional subspaces, these methods achieve significant memory reduction during pre-training. Extensions like RSO [ 11 ] provide theoretical convergence guarantees under low-rank gradient constraints. Advanced approaches such as Apollo [ 66 ] demonstrate additional benefits by combining channel-wise adaptive learning with low-rank gradients, yielding both improved validation accuracy and optimized memory efficiency throughout the pre-training process. Limitations in existing literature. Despite their computational and storage efficiency, current approaches utilizing low-rank LLM structures suffer from several critical limitations: (L1). Suboptimal performance with low-rank parameter training. Low-rank parameterization reduces memory and computational costs at the expense of suboptimal performance [ 4 ] . Techniques such as full-rank initialization [ 26 ] , update aggregation [ 29 , 37 ] , and non-linear operators [ 42 ] can mitigate this issue but may diminish computational benefits. Furthermore, recent studies show transformer weights typically exhibit near-full-rank properties [ 1 , 62 , 35 ] , with higher-rank weights being essential for knowledge representation [ 44 ] and sensitive to low-rank approximation [ 30 ] , indicating low-rank parameterization limit LLM capacity. (L2). Computational bottlenecks with low-rank gradient training. While training approaches utilizing low-rank gradient empirically achieve better performance than low-rank parameterization, the gradient compression process itself introduces computational overhead. Approaches such as GaLore [ 64 ] and FIRA [ 8 ] exploits Singular Value Decomposition (SVD) to identify effective low-rank gradient subspace, which would substantially reduce throughput during training. Other methods like GoLore [ 23 ] and RSO [ 11 ] propose to use random low-rank gradient subspace, which sometimes leads to suboptimal performance. (L3). Limitations to save activation memory. Existing approaches reduce memory load across multiple dimensions: parameters, gradients, and optimizer states. Yet a critical yet often overlooked memory burden stems from activation storage – intermediate variables cached during forward propagation required for gradient computation. Empirical studies [ 64 , 19 ] reveal that activation memory overhead typically ranges from 1× to 4× the model parameter size, exhibiting strong dependence on batch size configurations. While low-rank methods such as RSO [ 11 ] and CoLA-M [ 42 ] have been proposed to reduce activation memory, it remains an open question whether activations still possess potential for further compression. Main results. Our contributions are threefold, forming a principled framework rather than an ad-hoc combination: (C1). Novel Foundational Principle: We propose a novel principle in LLMs: the difference between activations of adjacent layers exhibits a strong low-rank structure. This phenomenon has been consistently observed across various models and at different training stages. Unlike low-rank properties in gradient updates or parameter modifications—such as those exploited by GaLoRE and LoRA—this structural property of activations represents an previously unreported characteristic of LLMs. It thereby serves as a foundational basis for more efficient pre-training paradigms. (C2). Parameter-efficient framework: We propose the C ross-layer low- R ank residual Net work ( CR-Net ), a pre-training framework that inherently utilizes cross-layer low-rank activation differences. By computing each linear layer’s activation through a combination of low-rank outputs and preceding layer activations, our design directly applies the observed low-rank structure while avoiding information loss from repeated low-rank approximations in LoRA-based approaches. This addresses Limitations (L1) and (L2) through reduced parameter complexity and memory/computation costs. (C3). Activation-efficient Re-computation: we develop a re-computation strategy for CR-Net that removes activation storage for most layers during backward propagation. Our analysis shows superior memory efficiency and lower re-computation overhead compared to existing methods like vanilla gradient checkpointing (GCP) and CoLA-M, resolving Limitation (L3) . (C4). Empirical Validation: Large-scale pre-training experiments demonstrate that CR-Net achieves better validation performance than existing low-rank parameter methods while maintaining training throughput. Integration with our re-computation strategy further reduces memory consumption without compromising model capability. Figure 1: Illustration of CR-Net base on LLaMA-2 architecture with L transformer layers. Layer normalization and ROPE are omitted for simplicity. 2 Preliminary and Related Works 2.1 Preliminary Notations. In this paper, we assume that the amount of transformer blocks of the model is L . Denote s and h as the sequence length and hidden dimension, respectively. h_{\text{ff}}

Read the complete text