From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models

Longwei Ding, Anhao Zhao Fanghua Ye, Ziyang Chen, Xiaoyu Shen · 2026

Read From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models by Longwei Ding, Anhao Zhao Fanghua Ye, Ziyang Chen, Xiaoyu Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18091.txt.

Introduction

Longwei Ding and co-authors ask whether pruning lessons from chat models carry over to reasoning models. Their 2026 preprint deals with model pruning, the cutting of parts to cut deployment cost. Most pruning studies target instruction-following models. Reasoning models differ: they write long chains of thought before they answer. It is unclear whether the old pruning rules still hold when the output is a long trace.

The team rethinks pruning for these reasoning-centric models. Which parts can go without harming the chain of thought is the key question. Cutting the wrong parts may save size but break reasoning. Their findings guide cheaper serving of the models now in demand for math and code.

Catalog record

CREATORLongwei Ding, Anhao Zhao Fanghua Ye, Ziyang Chen, Xiaoyu Shen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18091

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models Longwei Ding, Anhao Zhao Fanghua Ye, Ziyang Chen, Xiaoyu Shen Abstract Large language models (LLMs) are increasingly costly to deploy, motivating extensive research on model pruning. However, most existing studies focus on instruction-following LLMs, leaving it unclear whether established pruning strategies transfer to reasoning-augmented models that explicitly generate long intermediate reasoning traces. In this work, we conduct a controlled study of pruning for both instruction-following ( LLM-instruct ) and reasoning-augmented ( LLM-think ) models. To isolate the effects of pruning, we align pruning calibration and post-pruning recovery data with each model’s original training distribution, which we show yield more stable and reliable pruning behavior. We evaluate static depth pruning, static width pruning, and dynamic pruning across 17 tasks spanning classification, generation, and reasoning. Our results reveal clear paradigm-dependent differences: depth pruning outperforms width pruning on classification tasks, while width pruning is more robust for generation and reasoning. Moreover, static pruning better preserves reasoning performance, whereas dynamic pruning excels on classification and generation but remains challenging for long-chain reasoning. These findings underscore the need for pruning strategies that explicitly account for the distinct characteristics of reasoning-augmented LLMs. Our code is publicly available at https://github.com/EIT-NLP/LRM-Pruning . 1 Introduction Large language models (LLMs) have led to significant advances in natural language processing and enabled impactful applications across a wide range of domains 62 ; 52 ; 32 ; 1 ; 27 ; 16 . However, their strong empirical performance comes with substantial computational costs, posing major challenges for practical use in resource-constrained environments ( 60 ; 58 ) . Figure 1: Overview of the three structured pruning strategies. Static depth pruning removes entire layers, static width pruning reduces hidden dimensions (neurons or attention heads), and dynamic depth pruning adaptively skips layers, attention blocks, or MLP modules depending on the input. Model pruning has emerged as one of the most widely adopted approaches for improving LLM efficiency 43 ; 28 ; 30 . Early work has primarily focused on static pruning , where a fixed subset of parameters is removed prior to deployment. Within this paradigm, static depth pruning reduces inference cost by removing entire layers, shortening the model and decreasing sequential computation ( 30 ; 21 ) . In contrast, static width pruning reduces per-layer computation by shrinking hidden dimensions or pruning channels while preserving the original network depth ( 28 ; 4 ) . More recently, dynamic pruning has been explored, which adaptively skips parts of the computation during inference based on input-dependent signals, rather than permanently removing parameters ( 34 ; 20 ; 58 ) . Together, these techniques form a mature toolkit for accelerating LLMs. Meanwhile, the LLM landscape is undergoing a paradigm shift from instruction-following models, which we refer to as LLM-instruct , to reasoning-augmented models, denoted as LLM-think 51 ; 11 . Unlike LLM-instruct ( 54 ) , which generate a direct response with reasoning implicitly embedded, LLM-think explicitly externalize their reasoning process by producing step-by-step intermediate traces prior to the final answer ( 49 ) . While this shift substantially improves performance on complex tasks, it also results in markedly longer generations, often spanning thousands of tokens ( 7 ; 53 ) . Despite these fundamental changes in model behavior and output characteristics, existing pruning research has almost exclusively focused on LLM-instruct. This gap motivates a central question: does pruning for LLM-think require new strategies, or can existing approaches generalize across paradigms? Addressing this question necessitates a carefully controlled pruning pipeline. We therefore begin by revisiting a critical yet often overlooked design choice in prior work: the selection of datasets used for pruning calibration and post-pruning recovery . However, prior studies adopt highly heterogeneous datasets for these stages, ranging from general-domain corpora to instruction-tuning data, often without systematic justification ( 33 ; 44 ) . Such inconsistency introduces confounding factors, making it difficult to disentangle the effects of pruning from those of distribution mismatch. To enable a principled comparison, we adopt a controlled setting in which calibration and recovery data are strictly aligned with the original training distribution of the target model. Under this setting, we construct matched model–dataset pairs for both paradigms. For LLM-instruct , we use the open-sourced Tulu model ( 22 ) together with its full instruction-following fine-tuning corpus. For LLM-think , we derive a reasoning-augmented counterpart by further fine-tuning the same base model on OpenThoughts ( 17 ) , a dataset designed to elicit explicit multi-step reasoning across diverse difficulty levels. This alignment ensures that each model is pruned, calibrated, and recovered using data drawn from its own training distribution, thereby isolating the impact of pruning itself. As we show empirically, such alignment best preserves pre-pruning performance and leads to substantially more stable and effective pruning behavior compared to using mismatched datasets. Under this controlled setting, we conduct a comprehensive evaluation of three categories of pruning strategies on both LLM-instruct and LLM-think across 17 diverse classification, generation, and reasoning tasks. Our experiments yield several key empirical findings. First, static depth pruning is more effective than static width pruning on classification tasks, whereas the opposite trend holds for generation and reasoning tasks, where width pruning degrades performance more gracefully. Second, dynamic pruning with unconstrained token-level routing benefits classification and generation but proves substantially harder to optimize for long-chain reasoning, suggesting that aggressive computation skipping is particularly risky for reasoning-intensive workloads. Finally, while dynamic pruning generally outperforms static methods on classification and generation, static pruning retains higher performance on reasoning tasks, highlighting an important gap in current dynamic pruning approaches. 2 Background LLM-instruct and LLM-thinking The rapid progress of LLMs has not only improved their instruction-following ability but also expanded their scope toward explicit reasoning. This shift has given rise to two major families: instruction-following models ( LLM-instruct ), which are optimized to produce outputs that closely align with user-specified instructions, and reasoning-augmented models ( LLM-think ), which are trained to generate long chains of thought (CoT) reasoning before arriving at final answers. Although the two are built upon the same underlying architecture, recent studies show that they diverge significantly in both internal representations and emergent behaviors. Mechanistically, LLM-think retain richer contextual information across layers and exhibit higher token-level entropy ( 40 ; 47 ) . In terms of attention, LLM-instruct generally display diverse specialization across heads, with each attending to different token subsets ( 15 ) , whereas LLM-think show considerable overlap in tokens attended by different heads ( 55 ) . Formalizing Structure Pruning Strategies Model pruning in LLMs can be broadly categorized into two approaches: unstructured pruning , which removes individual weights based on their magnitudes or importance ( 23 ) , and structured pruning , which discards entire coupled structures ( 9 ) . Although unstructured pruning is conceptually simple, it rarely yields practical acceleration on modern GPUs (e.g., Nvidia GPUs typically require over 90% sparsity for speedup ( 39 ; 48 ) ), while LLMs usually collapse once sparsity exceeds 50% ( 42 ) . Therefore, we focus on structured pruning, which is both hardware-friendly and effective. Structured pruning can be further categorized into width pruning , which reduces hidden dimensions, and depth pruning , which removes redundant layers either statically

Read the complete text