Liger: Linearizing Large Language Models to Gated Recurrent Structures

Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du, Yu Cheng · 2025

Read Liger: Linearizing Large Language Models to Gated Recurrent Structures by Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du, Yu Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01496.txt.

Introduction

Disen Lan and four co-authors present Liger in 2025, a method that rewires big language models into leaner recurrent forms. Recurrent designs train in linear time and use fixed memory, but training them from scratch is costly and risky. Converting a trained transformer is cheaper, if the skill survives.

Older converters added new parts that needed heavy retraining and skipped the gating tricks of modern recurrent models. Liger adds nothing: it recycles the pretrained key weights to build gates. Light fine-tuning then restores full skill. A hybrid attention variant recovered ninety-three percent of the original model's quality for a sliver of the usual training cost.

Catalog record

CREATORDisen Lan, Weigao Sun, Jiaxi Hu, Jusen Du, Yu Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01496

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Liger: Linearizing Large Language Models to Gated Recurrent Structures Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du, Yu Cheng Abstract Transformers with linear recurrent modeling offer linear-time training and constant-memory inference. Despite their demonstrated efficiency and performance, pretraining such non-standard architectures from scratch remains costly and risky. The linearization of large language models (LLMs) transforms pretrained standard models into linear recurrent structures, enabling more efficient deployment. However, current linearization methods typically introduce additional feature map modules that require extensive fine-tuning and overlook the gating mechanisms used in state-of-the-art linear recurrent models. To address these issues, this paper presents Liger , short for Li nearizing LLMs to g at e d r ecurrent structures. Liger is a novel approach for converting pretrained LLMs into gated linear recurrent models without adding extra parameters. It repurposes the pretrained key matrix weights to construct diverse gating mechanisms, facilitating the formation of various gated recurrent structures while avoiding the need to train additional components from scratch. Using lightweight fine-tuning with Low-Rank Adaptation (LoRA), Liger restores the performance of the linearized gated recurrent models to match that of the original LLMs. Additionally, we introduce Liger Attention, an intra-layer hybrid attention mechanism, which significantly recovers 93% of the Transformer-based LLM at 0.02% pre-training tokens during the linearization process, achieving competitive results across multiple benchmarks, as validated on models ranging from 1B to 8B parameters. Figure 1: Liger Performance and Efficiency. Our proposed Liger recovers nearly 93% performance of Llama-3.2-1B and outperforms pretrained gated recurrent models at only 0.02% of the pre-training tokens cost. 1 Introduction Large language models (LLMs) have demonstrated exceptional performance across various natural language processing tasks ( Chintala, 2023 ; Team, 2023 ; Zhu et al., 2024 ; Qu et al., 2024 ) . However, the Transformer-based architecture ( Vaswani et al., 2017 ) used in modern LLMs, with its reliance on softmax attention, suffers from quadratic computational complexity. This inefficiency results in significant speed and memory challenges, particularly during pretraining on long sequences. During inference, the Key-Value (KV) cache ( Kwon et al., 2023 ) grows linearly with the input sequence length, leading to reduced inference speed and high memory usage, which severely limits the capability of these models for handling long-sequence tasks ( Sun et al., 2024a ) . In contrast, models based on linear recurrent modeling ( Katharopoulos et al., 2020 ; Yang et al., 2023 ; Qin et al., 2024b ; Sun et al., 2025 ; Du et al., 2025 ) provide linear-time training and constant-memory inference, offering substantial efficiency benefits and positioning themselves as promising candidates for the next generation of foundational architectures ( MiniMax et al., 2025 ) . † † footnotetext: The source code is available at https://github.com/OpenSparseLLMs/Linearization and the models are available at https://huggingface.co/collections/linear-moe-hub . While pretraining LLMs using architectures based on linear recurrent modeling reduces costs due to their linear training complexity, the high expenses of pretraining from scratch associated with large model sizes and datasets still remain a major obstacle to their adoption and practical use. This challenge has hindered the advancement of linear recurrent models. Linearizing pretrained LLMs like SUPRA ( Mercat et al., 2024 ) , MambaInLlama ( Wang et al., 2024 ) and LoLCATs ( Zhang et al., 2024a ) , as an emerging new direction, allows the transfer of weights from an existing pretrained model to one with linear recurrent modeling architectures at a small fraction of the original pretraining cost. The linearization approach is a promising post-training technique to enable efficient pretrained model deployment while preserving their performance. Gating mechanisms ( Qin et al., 2024a ; Sun et al., 2023 ) play a crucial role in linear recurrent models by controlling memory retention and forgetting, with their effectiveness widely demonstrated in such architectures. However, incorporating gate modules as additional components requires both transferring weights from pre-trained LLMs and training these gating modules from scratch. This process not only increases the cost of linearization but also creates a larger architectural divergence from Transformer-based LLMs. This divergence may hinder the effective approximation of softmax attention, limiting the performance of gated linear recurrent models ( Zhang et al., 2024d ) . Moreover, existing linearization methods often overlook the detailed design considerations of gated linear models, and the newly added modules fail to leverage the pre-trained weights of LLMs, further reducing the efficiency of linearization. In this paper, we present Liger , which stands for Li nearizing large language models to g at e d r ecurrent structures, a novel approach for linearizing LLMs. Liger repurposes the weights from pre-trained Transformer-based LLMs and introduces a novel method for constructing crucial gating mechanisms in gated recurrent structures using the key projection. This approach avoids the complex attention transfer process found in existing linearization methods. After transforming the weights and constructing the gating mechanisms, Liger requires only lightweight fine-tuning of the linearized gated recurrent model parameters through LoRA autoregressive training. By introducing Liger Attention, this efficient process restores further improves the model’s performance with minimal linearization cost, achieving competitive results across a range of language modeling and understanding benchmarks while benefiting from the linear-time inference efficiency of the recurrent architecture. Our contributions can be summarized as follows: • We introduce Liger , a novel method for adapting pre-trained Transformer-based LLMs into gated recurrent structures. This approach efficiently repurposes redundant weights from pre-trained models to construct gating modules without introducing additional parameters, obtaining gated recurrent LLMs with the benefits of constant-memory inference. • We propose Liger Attention , an intra-layer hybrid attention mechanism that combines sliding window softmax attention with linear recurrent modeling. This simple yet effective design retains the essential softmax non-linearity, accelerating the linearization process while maintaining the capabilities of pre-trained LLMs and ensuring linear-time inference efficiency. • We apply Liger to linearize the latest Llama-3 series, ranging from 1B to 8B parameters. Experimental results show that Liger outperforms existing linearization methods (like SUPRA ( Mercat et al., 2024 ) , MambaInLlama ( Wang et al., 2024 ) and LoLCATs ( Zhang et al., 2024a ) ), in terms of both efficiency and its ability to preserve the original performance of pre-trained LLMs. 2 Preliminary Transformer with Softmax Attention. Given the input sequence \mathbf{X}=\{\boldsymbol{x_{1}},\boldsymbol{x_{2}},\dots,\boldsymbol{x_{T}}\}\in\mathbb{R}^{T\times D} , with sequence length T and dimension D , vanilla transformer ( Vaswani et al., 2017 ) adopts standard softmax attention: \displaystyle\mathbf{Q},\mathbf{K},\mathbf{V} \displaystyle=\mathbf{X}\mathbf{W_{Q}},\mathbf{X}\mathbf{W_{K}},\mathbf{X}\mathbf{W_{V}} (1) \displaystyle\mathbf{O} \displaystyle=\operatorname{Softmax}((\frac{\mathbf{Q}\mathbf{K^{\top}}}{\sqrt{D}})\odot\mathbf{M})\mathbf{V} where \mathbf{W_{Q}} , \mathbf{W_{K}} , \mathbf{W_{V}}\in\mathbb{R}^{D\times D} are learnable parameters for input sequence \mathbf{X} projection and \mathbf{M}\in\mathbb{R}^{T\times T} is a mask matrix for causal modeling by preventing future information leakage in autoregressive generation task. The above parallel form of softmax attention in Eq. 1 is applied for efficient training and can be rewritten in the following recurrent form during inference stage: \displaystyle\boldsymbol{q}_{t},\boldsymbol{k}_{t},\boldsymbol{v}_{t} \displaystyle=\boldsymbol{x}_{t}\mathbf{W_{Q}},\boldsymbol{x}_{t}\mathbf{W_{K}},\boldsymbol{x}_{t}\mathbf{W_{V}} (2) \displaystyle\boldsymbol{o}_{t} \displaystyle=\frac{\sum_{i=1}^{t}\operatorname{exp}(\boldsymbol{q}_{t}\boldsymbol{k}_{i}^{\top}/\sqrt{D})\boldsymbol{v}_{i}}{\sum_{i=1}^{t}\operatorname{exp}(\boldsymbol{q}_{i}\boldsymbol{k}_{i}^{\top}/\sqrt{D})} The standard softmax attention is highly reliant on the growing KV Cache ( Chou et al., 2024b ; Wang et al., 2024 ) to recall the history "memory" for sequence modeling, which results in quadratic complexity and costly memory requirements especially in long context setting. Linear Attention. Linear transformer ( Katharopoulos et al., 2020 ; Qin et al., 2023 ) approximates softmax self-attention as the dot product of the kernel feature mapping and utilizes associative property of

Read the complete text