Lizard: An Efficient Linearization Framework for Large Language Models

Chien Van Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A. Rossi, Trung Bui, Nikos Vlassis, Franck Dernoncourt, Thien Huu Nguyen · 2025

Read Lizard: An Efficient Linearization Framework for Large Language Models by Chien Van Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A. Rossi, Trung Bui, Nikos Vlassis, Franck Dernoncourt, Thien Huu Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-09025.txt.

Introduction

Chien Van Nguyen and co-authors present Lizard, a method that converts large language models into faster, leaner designs, in a 2025 paper. Transformer models slow down badly on long inputs. Their attention cost grows with the square of length. The key-value cache also swells until memory chokes. Linearized models promise relief but often lose quality.

Lizard swaps in a subquadratic attention that closely mimics the original. Unlike fixed earlier schemes, its structure adapts to the model it converts. Pretrained models keep their quality after conversion. Long documents and chats become cheaper to serve. The framework suits firms that must cut inference cost at scale. It also opens research on hybrid attention designs.

Catalog record

CREATORChien Van Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A. Rossi, Trung Bui, Nikos Vlassis, Franck Dernoncourt, Thien Huu Nguyen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-09025

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lizard: An Efficient Linearization Framework for Large Language Models Chien Van Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A. Rossi, Trung Bui, Nikos Vlassis, Franck Dernoncourt, Thien Huu Nguyen Abstract We propose Lizard, a linearization framework that transforms pretrained Transformer-based Large Language Models (LLMs) into flexible, subquadratic architectures for infinite-context generation. Transformer-based LLMs face significant memory and computational bottlenecks as context lengths increase, due to the quadratic complexity of softmax attention and the growing key-value (KV) cache. Lizard addresses these limitations by introducing a subquadratic attention mechanism that closely approximates softmax attention while preserving the output quality. Unlike previous linearization methods, which are often limited by fixed model structures and therefore exclude gating mechanisms, Lizard incorporates a gating module inspired by recent state-of-the-art linear models. This enables adaptive memory control, supports constant-memory inference, offers strong length generalization, and allows more flexible model design. Lizard combines gated linear attention for global context compression with sliding window attention enhanced by meta memory, forming a hybrid mechanism that captures both long-range dependencies and fine-grained local interactions. Moreover, we introduce a hardware-aware algorithm that accelerates the training speed of our models. Extensive experiments show that Lizard achieves near-lossless recovery of the teacher model’s performance across standard language modeling tasks, while significantly outperforming previous linearization methods. On the 5-shot MMLU benchmark, Lizard improves over prior models by 18 points and shows significant improvements on associative recall tasks. 1 Introduction Large Language Models (LLMs) based on Transformer architectures have achieved impressive advances in natural language processing across various tasks ( Grattafiori et al. 2024 ; Mann et al. 2020 ; Achiam et al. 2023 ) . However, their reliance on softmax attention with quadratic time and memory complexity poses significant limitations for long-context applications, both during training and inference. In particular, the softmax attention computation scales quadratically with the sequence length, while the key-value (KV) cache grows linearly during generation, resulting in significant computational and memory overhead for long-context sequences ( Adnan et al. 2024 ) . Recent work has proposed linear and subquadratic alternatives to softmax attention ( Yang et al. 2023 ; Gu and Dao 2023 ; Peng et al. 2023b ; Wang et al. 2020 ; Dao and Gu 2024 ) , enabling linear-time training and constant-memory inference. However, despite these efficiency benefits, pretraining LLMs with such architectures from scratch requires massive training budgets, often involving trillions of tokens. More importantly, models trained with linear attention mechanisms from scratch consistently underperform on tasks that require in-context learning and retrieval. For example, Transformer models significantly outperform both Mamba and Mamba-2 by up to 15 points on the 5-shot MMLU benchmark when all are pretrained under the same settings ( Waleffe et al. ) . Another promising direction involves linearizing pretrained Transformer-based LLMs by replacing their softmax attention modules with subquadratic alternatives ( Zhang et al. ; Mercat et al. 2024 ; Zhang et al. 2024 ; Wang et al. 2024 ) . Instead of training new models from scratch with linear attention, this approach leverages the capabilities of openly available, large-scale pretrained models typically trained on trillions of tokens, and retrofits them with more efficient attention mechanisms. By starting from high-quality pretrained models, linearized models can better preserve task performance and recover softmax-like spiky attention patterns, thereby enhancing retrieval performance in in-context learning tasks ( Zhang et al. 2024 ) . However, prior work either fails to fully exploit the strengths of linear attention mechanisms or still significantly underperforms compared to softmax-based attention. For example, LoLCATs ( Zhang et al. 2024 ) , a recent state-of-the-art linearization method, shows promising results and cost-effectiveness. However, it still relies on predefined positional embedding patterns for positional encoding, thereby struggling with length generalization and failing to extrapolate beyond the trained context length. Moreover, due to architectural constraints and limited flexibility, it lacks an efficient memory update mechanism, which significantly affects its performance on recall-intensive tasks. On the other hand, models such as MambaInLlama ( Wang et al. 2024 ) show better extrapolation to longer sequences, but still suffer from substantial performance degradation. Specifically, on the 5-shot MMLU benchmark, LoLCATs lags behind its teacher model by 13.8 points, while MambaInLLaMA exhibits a 23.4 point drop. In this paper, we introduce Lizard ( Li neari z ing Softmax A ttention with R ecurrent Gate D ynamics), an efficient framework for linearizing LLMs. Unlike prior approaches that constrain themselves to reusing only the pretrained model’s existing parameters and fixed architecture ( Mercat et al. 2024 ; Zhang et al. 2024 ) , we introduce additional learnable components in the form of a gating module. The gating module serves two primary purposes. First, it acts as a data-dependent alternative to predefined positional embedding methods, such as Rotary Positional Embeddings (RoPE) ( Heo et al. 2024 ) , allowing the model to learn implicit relative positional information through adaptive decay patterns, enhancing length generalization. Second, the gated recurrent structure provides a dynamic memory management mechanism, enabling the model to control the retention and forgetting of past tokens during generation, thereby improving associative recall. To further recover the expressiveness of softmax attention, we combine the output of the gated linear attention with a sliding window attention mechanism augmented with meta memory. While the gated component captures global context in a compressed form, the sliding window attention preserves local precision and expressivity within the local context. This hybrid attention framework provides a high-quality approximation to softmax attention. We train our model in two stages: (1) an attention approximation stage where the subquadratic modules are trained to mimic softmax attention outputs, and (2) a fine-tuning stage where the linearized model is adapted to downstream language modeling objectives. Our contributions can be summarized as follows: • We propose Lizard, a linearization framework that adapts pretrained Transformer-based LLMs into gated recurrent architectures. This adaptation enables a highly efficient, subquadratic attention mechanism with constant-memory inference and strong length generalization. Lizard achieves near-lossless recovery of the original model’s performance across major benchmarks and outperforms prior linearization methods by 8–18 points on the 5-shot MMLU benchmark. • We conduct extensive empirical studies to analyze architectural design choices within Lizard and evaluate which state-of-the-art recurrent architectures are most effective, across diverse benchmarks. • We present a hardware-aware implementation that improves the training throughput by up to 24%, enabling faster optimization. 2 Preliminary To motivate Lizard, we first review the core components of Causal Softmax Attention, Gated Linear Attention, and techniques for approximating softmax attention. We then briefly discuss related work on linearizing Transformers and adapting pretrained LLMs for efficient long-context modeling. Causal Softmax Attention In modern Transformer architectures ( Touvron et al. 2023 ; Jiang et al. ) , for a query vector at position i , Causal Softmax Attention with RoPE produces the output \mathbf{y}_{i} as: \mathbf{y}_{i}=\sum_{t=1}^{i}\frac{\exp\left(\varphi_{\text{RoPE}}(\mathbf{q}_{i})^{\top}\varphi_{\text{RoPE}}(\mathbf{k}_{t})/\sqrt{d}\right)}{\sum_{j=1}^{i}\exp\left(\varphi_{\text{RoPE}}(\mathbf{q}_{i})^{\top}\varphi_{\text{RoPE}}(\mathbf{k}_{j})/\sqrt{d}\right)}\mathbf{v}_{t}, where \varphi_{\text{RoPE}}(\mathbf{q}_{i}) and \varphi_{\text{RoPE}}(\mathbf{k}_{i}) denote RoPE-transformed query and key vectors, respectively. The causal softmax attention with RoPE can be expressed in matrix multiplication form as below: \mathbf{Y}_{\text{softmax}}=\mathrm{softmax}\left(\mathbf{Q}_{\text{RoPE}}\mathbf{K}_{\text{RoPE}}^{\top}/{\sqrt{d}\odot\mathbf{M}}\right)\mathbf{V} where L is the context length, \mathbf{Q}_{\text{RoPE}},\mathbf{K}_{\text{RoPE}}\in\mathbb{R}^{L\times d} are the RoPE-transformed query and key matrices, M\in\mathbb{R}^{L\times L} is the lower-triangular causal mask ( \mathbf{M}_{ij}=1 if i\geq j , and \mathbf{M}_{ij}=-\infty if i<j ), \mathbf{V}\in\mathbb{R}^{L\times d} is the value

Read the complete text