Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization
Hanyu Li †§ Jiangshan Duo †‡ Bofei Gao †‡ Liang Zhao † Hailin Zhang † Sujian Li ‡ Xiaotie Deng § Fuli Luo † § CFCS, School of Computer Science, Peking University ‡ State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University † LLM-Core Xiaomi · 2026
Read Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization by Hanyu Li †§ Jiangshan Duo †‡ Bofei Gao †‡ Liang Zhao † Hailin Zhang † Sujian Li ‡ Xiaotie Deng § Fuli Luo † § CFCS, School of Computer Science, Peking University ‡ State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University † LLM-Core Xiaomi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06052.txt.
Introduction
Long chains of thought make language models costly and slow. Hanyu Li and 6 co-authors tackle that overthinking trap in a 2026 paper. Fixed limits on reasoning length can cut steps a problem actually needs. Their method shortens reasoning only when shortening is safe.
The approach uses reinforcement learning with a mastery gate at the level of each sample. Long answers are penalized only when the model already solves the problem and has produced a shorter solution. Response length fell by a fifth to two-fifths with accuracy kept or improved. Training on math alone shortened unseen tasks like code, and gains transferred both ways between plain reasoning and tool-using agents.
Catalog record
| CREATOR | Hanyu Li †§ Jiangshan Duo †‡ Bofei Gao †‡ Liang Zhao † Hailin Zhang † Sujian Li ‡ Xiaotie Deng § Fuli Luo † § CFCS, School of Computer Science, Peking University ‡ State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University † LLM-Core Xiaomi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-06052
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization Hanyu Li †§ Jiangshan Duo †‡ Bofei Gao †‡ Liang Zhao † Hailin Zhang † Sujian Li ‡ Xiaotie Deng § Fuli Luo † § CFCS, School of Computer Science, Peking University ‡ State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University † LLM-Core Xiaomi Abstract Chain-of-thought reasoning in large language models can trigger an “overthinking trap”: longer rollouts raise cost and latency yet often yield unreliable accuracy gains. Existing methods use global, static controls that may suppress needed reasoning. We propose mastery-gated, sample-level, soft reinforcement learning compression that penalizes long rollouts only when the model already solves the problem and has produced a shorter rollout. Across benchmarks, it cuts response length by 20-40% with comparable or higher accuracy and generalizes across domains: a model trained on math spontaneously shortens unseen tasks (code, instruction following, general-knowledge QA) without hurting accuracy. We further show two-way transfer between non-agent CoT and tool-use agents: non-agent training reduces SWE-Bench Verified rounds by 13%, while compressing a thinking agent cuts SWE trajectories by 67% tokens and 52% rounds and shortens non-agent outputs by up to 44%. Compression is thus not cosmetic brevity, but an inherent computation policy — what to keep, and what to forget. \titlefont Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization Hanyu Li †§⋆ Jiangshan Duo †‡ Bofei Gao †‡ Hailin Zhang † Sujian Li ‡ Xiaotie Deng § Liang Zhao †⋄ § CFCS, School of Computer Science, Peking University ‡ State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University † LLM-Core Xiaomi \abscontent Figure 1 : Overview of our compression method across diverse tasks. Each panel shows one benchmark: the orange curve is response length (tokens, normalized to 100% at that benchmark’s starting point) and the blue curve is the core metric (accuracy or score), illustrating that compression training only on math problems broadly shortens chains of thought while maintaining or improving downstream performance. 0 0 footnotetext: ⋆ Contribution during internship at Xiaomi LLM-Core Team. 0 0 footnotetext: ⋄ Corresponding author. 1 Introduction Generating long chains of thought (CoT) has significantly improved the complex reasoning abilities of large language models ( 26 ; 5 ) . However, this success introduces an “overthinking trap,” characterized by high computational cost, increased latency, and verbose reasoning paths that do not always guarantee higher accuracy ( 23 ; 31 ) . We posit that true intelligence manifests as compression. Once a model masters a problem-solving path, a more efficient representation must exist. We argue that compression is not merely a trade-off for efficiency but an indicator of internalized, mature capability. Prevailing approaches, including reinforcement learning with various length penalties ( 23 ; 31 ) and various forms of distillation ( 7 ; 2 ; 11 ) , have laid a crucial foundation. They demonstrate the feasibility of compressing CoT and offer valuable mechanisms for controlling computational budgets, often achieving effective accuracy-efficiency trade-offs. However, these methods typically rely on global and static controls, such as a uniform penalty or a fixed truncation limit. Such “one-size-fits-all” policies risk operating at the wrong unit : they do not adapt to the heterogeneous difficulty of individual samples and may inadvertently penalize necessary, extended reasoning. Furthermore, they often apply compression at the wrong time , forcing shorter solutions before the model has reliably mastered the problem and causing the reward hacking problem. We contend that effective length control must be sample-level, gated, and soft, rather than global, ungated, and hard. We propose a minimal intervention: apply soft compression only to samples the model has already mastered, and only on reasoning paths that exceed a “‘safe length” derived from its own correct solutions. This approach ensures that difficulty, which we distinguish from length, is respected. A model must first learn to solve a problem reliably; only then is it encouraged to find a shorter path. We penalize only the excessive verbosity on problems the model demonstrably understands, rather than forcing truncation on problems it is still reasoning. Our experiments reveal several key findings. First, this sample-level, gated, soft compression framework significantly outperforms global penalties, while hard truncation methods fail catastrophically. The gating mechanism — compressing only after mastery is achieved — proves essential for maintaining robust performance. Second, this compression skill generalizes: models trained to compress reasoning on a specific domain (e.g., mathematics) spontaneously reduce verbosity across all tasks. Third, we identify a stable, multi-stage training recipe which can reduce average inference length by 20–40% with comparable or even improved accuracy, a robust result across model structures, scales, and tasks. Finally, we show that this learned notion of “efficient computation” extends beyond single-turn CoT: it transfers to tool-use agents and even exhibits two-way generalization between non-agent reasoning and agent trajectories. Based on these findings, we argue that reasoning compression should be elevated from an optimization trick to a standard, distinct phase of post-training. We propose a stable curriculum: first, enhance accuracy; second, apply safe, gated compression with early stopping; finally, conduct another accuracy-enhancing phase to recover or further boost performance. This transforms the principle of “compression as intelligence” into a repeatable engineering process, capable of significantly improving inference efficiency by learning when and how to compress. Our contributions are threefold. We (1) propose a minimal, stable, and sample-level gated compression paradigm, formalized as a practical post-training recipe; (2) systematically validate the mechanisms, from motivation to empirical phenomena, providing reproducible evidence of its dynamics, generalization, and failure modes, leading to a robust early stopping strategy; (3) reframe compression from a mere cost-optimization technique to a measurable process of capability internalization, and show it as an agent-agnostic efficiency skill that applies to both pure reasoning and tool-use settings. 2 Related Work Reinforcement learning for length control. Most prior work relies on global, static policies-hard truncation (ThinkPrune ( 6 ) ), truncation-penalized RL (DLER ( 13 ) ), length-harmonizing fine-tuning (O1-Pruner ( 16 ) ), constraint-following RL (L1 ( 1 ) ), sample-level non-gated length penalties RL in Kimi K1.5 ( 23 ) , and global non-gated length penalties in DAPO ( 31 ) — whereas we apply sample-level, gated, soft compression that triggers only after mastery and only beyond a self-derived safe length. Distillation, pruning, and model merging. CoT distillation (Distilling Step-by-Step ( 7 ) , EDIT ( 2 ) , D&R Distillation ( 11 ) ), token/computation pruning and compression (Efficient Long CoT ( 25 ) , DLCoT ( 17 ) , TokenSkip ( 28 ) , LightThinker ( 32 ) , Prune-on-Logic ( 33 ) ), and model merging (Thinking Spectrum ( 10 ) , model merging for length-controlled reasoning ( 27 ) ) pursue similar accuracy-efficiency goals, yet our loop is on-policy and compresses only when the model has demonstrably mastered the instance. 3 Method We adopt the standard reinforcement learning with verifiable rewards (RLVR) training strategy ( 5 ) . We compress only where the model already knows the answer. For each training sample, tokens generated beyond a per-sample “safe length” are softly penalized, but only after the model has demonstrated reliable mastery in solving that sample. Let x be an input problem, y be a full generated response