EcoSpa: Efficient Transformer Training with Coupled Sparsity

Jinqi Xiao, Cheng Luo, Lingyi Huang, Cheng Yang, Yang Sui, Huy Phan, Xiao Zang, Yibiao Ying, Zhexiang Tang, Anima Anandkumar, Bo Yuan · 2025

Read EcoSpa: Efficient Transformer Training with Coupled Sparsity by Jinqi Xiao, Cheng Luo, Lingyi Huang, Cheng Yang, Yang Sui, Huy Phan, Xiao Zang, Yibiao Ying, Zhexiang Tang, Anima Anandkumar, Bo Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11641.txt.

Introduction

Jinqi Xiao and ten co-authors introduce EcoSpa, a sparse training method for transformers that respects how weight matrices work in coupled pairs. Attention and feed-forward layers multiply matrices together, yet most sparsification methods prune each matrix alone and break those interactions, hurting quality at high sparsity.

EcoSpa judges and prunes coupled pairs together, removing aligned rows and columns so the interaction pattern survives. A new granularity for scoring structural importance supports both pretraining and fine-tuning runs. The payoff sought is real efficiency, training large models with far fewer active weights, without the accuracy collapse that naive pruning causes when sparsity grows aggressive.

Catalog record

CREATORJinqi Xiao, Cheng Luo, Lingyi Huang, Cheng Yang, Yang Sui, Huy Phan, Xiao Zang, Yibiao Ying, Zhexiang Tang, Anima Anandkumar, Bo Yuan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11641

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EcoSpa: Efficient Transformer Training with Coupled Sparsity Jinqi Xiao, Cheng Luo, Lingyi Huang, Cheng Yang, Yang Sui, Huy Phan, Xiao Zang, Yibiao Ying, Zhexiang Tang, Anima Anandkumar, Bo Yuan Abstract Transformers have become the backbone of modern AI, yet their high computational demands pose critical system challenges. While sparse training offers efficiency gains, existing methods fail to preserve critical structural relationships between weight matrices that interact multiplicatively in attention and feed-forward layers. This oversight leads to performance degradation at high sparsity levels. We introduce EcoSpa, an efficient structured sparse training method that jointly evaluates and sparsifies coupled weight matrix pairs, preserving their interaction patterns through aligned row/column removal. EcoSpa introduces a new granularity for calibrating structural component importance and performs coupled estimation and sparsification across both pre-training and fine-tuning scenarios. Evaluations demonstrate substantial improvements: EcoSpa enables efficient training of LLaMA-1B with 50% memory reduction and 21% faster training, achieves 2.2 × 2.2\times model compression on GPT-2-Medium with 2.4 2.4 lower perplexity, and delivers 1.6 × 1.6\times inference speedup. The approach uses standard PyTorch operations, requiring no custom hardware or kernels, making efficient transformer training accessible on commodity hardware. 1 Introduction Transformers 47 have become the dominant architecture for modern machine learning, powering breakthroughs in natural language processing 13 ; 38 ; 45 , computer vision 14 ; 30 , and beyond. However, their computational demands pose critical system challenges. Training large models like LLaMA-7B requires extensive GPU resources 45 , while deployment demands high-end accelerators, limiting accessibility to well-resourced institutions and increasing operational costs for practical applications. Sparse training has emerged as a promising solution to these challenges. By dynamically identifying and removing unimportant parameters during training, sparse methods can reduce computational and memory costs while maintaining model quality. As illustrated in Fig. 1 , sparse training applies to both pre-training scenarios (training from random initialization) and fine-tuning scenarios (adapting pre-trained models), offering efficiency improvements across the model lifecycle. Recent works have explored various structured sparsity approaches for transformers: head-level sparsity 5 ; 10 ; 31 , layer-wise removal 9 ; 33 , and fine-grained row/column pruning 4 ; 46 . Figure 1: Overview of EcoSpa. Unlike existing methods that independently sparsify weight matrices, EcoSpa jointly estimates the importance of coupled weight matrices and removes aligned row/column pairs, preserving coupling relationships. The approach is applicable to both pre-training and fine-tuning scenarios. The coupling problem. Despite progress in sparse training for transformers 5 ; 10 ; 4 ; 46 , most existing methods do not explicitly account for a critical architectural property: weight matrices in transformers often operate in coupled pairs whose outputs interact multiplicatively. For example, in multi-head attention, \bm{W}_{Q} and \bm{W}_{K} produce queries and keys whose interaction determines attention scores. Similarly, feed-forward networks use consecutive matrices \bm{W}_{1} and \bm{W}_{2} whose outputs must be dimensionally compatible. Current approaches 4 ; 46 typically sparsify these coupled matrices by treating each matrix as an isolated component, which can disrupt their interaction and lead to performance degradation, especially at high sparsity levels where the remaining parameters must collectively preserve the model’s representational capacity. This makes the preservation of their structural relationships critical for system efficiency. This paper introduces EcoSpa, a structured sparse training method that preserves coupling relationships during sparsification. Rather than independently removing rows/columns from individual matrices, EcoSpa jointly evaluates coupled matrix pairs and removes aligned row/column pairs—ensuring that when a column is removed from \bm{W}_{Q} , the corresponding column is also removed from \bm{W}_{K} . This alignment preserves the dimensionality and interaction of their outputs while reducing memory footprint and computational cost. We adapt empirical Fisher information to the context of coupled matrix pairs, enabling efficient identification of which pairs contribute least to model output, and then apply coordinated removal within each pair. Theoretical analysis (Appendix A ) shows this approach better preserves the approximation quality of matrix products compared to independent sparsification. Contributions and Results. We introduce two key innovations: ( i ) Coupled estimation that assesses importance at the granularity of coupled weight matrix pairs rather than individual matrices, and ( ii ) Coupled sparsification that removes aligned row/column pairs to preserve coupling relationships and dimensional compatibility. Our comprehensive evaluations demonstrate substantial system improvements: EcoSpa reduces GPU memory by 50% (3.88GB vs 7.80GB) and training time by 21% for LLaMA-1B pre-training, enabling efficient deployment while achieving 1.6\times inference speedup. For GPT-2-Medium, it delivers 2.2\times compression with 2.4 lower perplexity and 40% faster training. Vision Transformers achieve higher accuracy than prior sparse training methods at comparable compression levels. Ablation studies validate that coupled sparsification substantially outperforms independent approaches, with performance gaps widening at high sparsity levels where efficiency matters most for practical deployment. 2 Background 2.1 Preliminaries Notation. We represent tensors using boldface calligraphic script, denoted as \bm{\mathcal{X}} . Matrices and vectors are indicated with boldface capital and lowercase letters, such as \bm{X} and \bm{x} , respectively. Furthermore, non-boldface letters with indices, e.g. , \mathcal{X}(i_{1},\cdots,i_{d}) , X(i,j) , and x(i) , denote the entries for a d -dimensional tensor \bm{\mathcal{X}} , a matrix \bm{X} , and a vector \bm{x} , respectively. Transformer. For Transformer-based models, the key components include the Multi-Head Attention (MHA) and Feed-Forward Network (FFN). More specifically, the MHA operation is defined as follows: {\rm MHA}(\bm{X}_{Q},\bm{X}_{K},\bm{X}_{V})={\rm Concat}(head_{1},\dots,head_{h})\bm{W}^{O}, (1) where \bm{X}_{Q},\bm{X}_{K},\bm{X}_{V}\in\mathbb{R}^{l\times d_{m}} are the length- l input sequences, d_{m} is the embedding dimension, and h is the number of attention heads. Here each attention head head_{i} operates as below: \displaystyle head_{i}={\rm Attention}(\bm{X}_{Q}\bm{W}_{i}^{Q},\ \bm{X}_{K}\bm{W}_{i}^{K},\ \bm{X}_{V}\bm{W}_{i}^{V})=\phi\left(\frac{\bm{X}_{Q}\bm{W}_{i}^{Q}(\bm{X}_{K}\bm{W}_{i}^{K})^{\top}}{\sqrt{d_{h}}}\right)\bm{X}_{V}\bm{W}_{i}^{V}, (2) where \phi(\cdot) represents the softmax function, \bm{W}_{i}^{Q} , \bm{W}_{i}^{K},\bm{W}_{i}^{V}\in\mathbb{R}^{d_{m}\times d_{h}} , \bm{W}^{O}\in\mathbb{R}^{d_{m}\times d_{m}} , and d_{m}=d_{h}\times h . The FFN consists of two fully connected layers with a Gaussian Error Linear Unit (GELU) activation function applied in between. Let \bm{X} represent the input embeddings, and \bm{W}_{in}\in\mathbb{R}^{d_{m}\times d},\bm{W}_{out}\in\mathbb{R}^{d\times d_{m}},\bm{b}_{in},\bm{b}_{out} denote the weight matrices and bias vectors, respectively. The operation of FFN can be then defined as follows: {\rm FFN}(\bm{X})={\rm GELU}(\bm{X}\bm{W}_{in}+\bm{b}_{in})\bm{W}_{out}+\bm{b}_{out}. (3) 2.2 Related Work Sparse Training for Transformers. Sparse training for transformers aims to reduce computational and memory costs by identifying and removing unimportant parameters during training. Existing methods can be categorized by their target training stage and sparsification granularity. Pre-training from Scratch. Training sparse transformers from random initialization remains under-explored compared to sparse CNN pre-training, where dynamic sparse training methods 35 ; 36 ; 16 ; 58 ; 8 and structured sparsification 20 ; 49 ; 29 ; 21 have been extensively studied. For transformers, Chen et al. 5 dynamically extract and train sparse sub-networks for Vision Transformers through unstructured or structured sparsification, alleviating memory bottlenecks. Chen et al. 10 identify structured winning lottery tickets early in BERT pre-training to improve efficiency. Dao et al. 11 propose Monarch matrices—a class of structured matrices that approximate dense weights for hardware-efficient pre-training. Han et al. 18 combine low-rank and unstructured sparse matrices for the pre-training model. Low-rank compression with hardware awareness 54 and attention-centric customization frameworks 53 further demonstrate the benefits of coordinating structural assumptions across coupled transformer blocks. These methods reduce training costs but typically operate at coarse granularities (entire heads or layers) or apply sparsification independently to each weight matrix, with limited consideration of interactions between coupled matrices. Fine-tuning Pre-trained Models. Sparsifying pre-trained models for efficient deployment has attracted significant attention. Methods fall into

Read the complete text