The Overthinker’s DIET : Cutting Token Calories with DI fficulty - Awar E T raining

Weize Chen, Jiarui Yuan, Tailin Jin, Ning Ding, Huimin Chen, Zhiyuan Liu, Maosong Sun · 2025

Read The Overthinker’s DIET : Cutting Token Calories with DI fficulty - Awar E T raining by Weize Chen, Jiarui Yuan, Tailin Jin, Ning Ding, Huimin Chen, Zhiyuan Liu, Maosong Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19217.txt.

Introduction

Weize Chen and six co-authors put reasoning models on a diet, trimming the token bloat of overlong answers. Their framework, DIET, appeared in twenty twenty-five. Strong models often think far past the problem's needs, wasting compute on easy questions.

The training scheme estimates each problem's difficulty on the fly. Penalties for length then tighten or loosen to match, and target lengths follow the estimate. A weighting fix keeps the group-based training stable, which naive reward tweaks had broken. Results cut token counts while accuracy actually rose. Under fixed budgets, shorter answers let majority voting sample more tries, scaling better than rivals. Verbosity also lined up more honestly with true difficulty.

Catalog record

CREATORWeize Chen, Jiarui Yuan, Tailin Jin, Ning Ding, Huimin Chen, Zhiyuan Liu, Maosong Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19217

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Overthinker’s DIET : Cutting Token Calories with DI fficulty - Awar E T raining Weize Chen, Jiarui Yuan, Tailin Jin, Ning Ding, Huimin Chen, Zhiyuan Liu, Maosong Sun Abstract Recent large language models (LLMs) exhibit impressive reasoning but often overthink , generating excessively long responses that hinder efficiency. We introduce DIET ( DI fficulty-Awar E T raining) , a framework that systematically cuts these "token calories" by integrating on-the-fly problem difficulty into the reinforcement learning (RL) process. DIET dynamically adapts token compression strategies by modulating token penalty strength and conditioning target lengths on estimated task difficulty, to optimize the performance-efficiency trade-off. We also theoretically analyze the pitfalls of naive reward weighting in group-normalized RL algorithms like GRPO, and propose Advantage Weighting technique, which enables stable and effective implementation of these difficulty-aware objectives. Experimental results demonstrate that DIET significantly reduces token counts while simultaneously improving reasoning performance. Beyond raw token reduction, we show two crucial benefits largely overlooked by prior work: (1) DIET leads to superior inference scaling . By maintaining high per-sample quality with fewer tokens, it enables better scaling performance via majority voting with more samples under fixed computational budgets, an area where other methods falter. (2) DIET enhances the natural positive correlation between response length and problem difficulty, ensuring verbosity is appropriately allocated, unlike many existing compression methods that disrupt this relationship. Our analyses provide a principled and effective framework for developing more efficient, practical, and high-performing LLMs. 1 Introduction Recent breakthroughs in large language models (LLMs) have yielded remarkable reasoning capabilities, particularly when enhanced through reinforcement learning (RL) from outcome-based rewards ( OpenAI 2024 ; DeepSeek-AI et al. 2025 ; Team 2025 ) . These models excel in complex domains like mathematics and coding, often generating sophisticated reasoning chains ( Gandhi et al. 2025 ; Pan et al. 2025 ; Luo et al. 2025b ) . However, this enhanced reasoning frequently comes with a significant side effect: a dramatic increase in response length compared to base or instruction-tuned models. While some verbosity can facilitate complex thought, it often leads to overthinking : models produce excessively long responses, sometimes thousands of tokens, even for simple queries (e.g., "2+3=?") ( Chen et al. 2024 ; Chang et al. 2025 ; Luo et al. 2025a ) . This verbosity severely impacts inference latency and computational costs, hindering the practical deployment of these powerful reasoning models. Initial attempts to mitigate overthinking via supervised fine-tuning (SFT), direct preference optimization (DPO), or simple length penalties in RL objectives ( Team et al. 2025 ; Xia et al. 2025 ; Aggarwal & Welleck 2025 ) often struggle, leading to performance degradation, i.e., models that are concise but inaccurate. A crucial dimension often overlooked in token compression is the intrinsic link between problem difficulty and the appropriate level of verbosity. We contend that a "one-size-fits-all" compression strategy is fundamentally flawed. Complex problems may necessitate longer, more detailed reasoning, whereas simpler ones should elicit direct, concise answers. Indeed, as we later show ( Section 2.3 ), LLMs often naturally exhibit a tendency to use more tokens for problems they find more challenging. Token compression methods that ignore this inherent relationship could force a suboptimal compromise across the difficulty spectrum. We argue that explicitly incorporating on-the-fly difficulty estimation is essential for developing token compression strategies that effectively preserve reasoning capabilities. To address these challenges, we introduce DIET ( DI fficulty-Awar E T raining) , a framework to systematically "cut token calories" from overthinking LLMs. DIET integrates on-the-fly problem difficulty into the RL training process, dynamically adapting token compression based on estimated task difficulty. This enables DIET to selectively encourage conciseness for simpler problems while preserving necessary verbosity for complex ones, improving the performance-efficiency trade-off and achieving better reasoning with significantly fewer tokens than the base model. The effective implementation of such difficulty-aware objectives within popular RL algorithms like GRPO ( Shao et al. 2024 ) presents its own challenges. We theoretically analyze the pitfalls of naively applying weighted rewards in these settings and propose a robust Advantage Weighting technique that ensures stable and effective token compression training. Beyond standard performance gains, our work reveals two often-overlooked benefits of difficulty-aware compression: (1) Critically for practical deployment, we show that DIET leads to superior inference scaling . By maintaining high per-sample quality with fewer tokens, it enables significantly better inference scaling performance under small computational budgets, an area where most prior token compression efforts falter or show degradation. (2) We demonstrate that DIET not only compresses, but also enhances the natural positive correlation between an LLM’s response length and problem difficulty: a desirable characteristic for adaptive reasoning that many existing compression methods inadvertently disrupt. This paper thus formalizes the difficulty-aware token compression problem and presents DIET as a comprehensive solution. Through the core adaptive mechanisms, the enabling Advantage Weighting technique, and by demonstrating unique benefits in enhancing inference scaling and preserving adaptive verbosity, DIET offers a principled and highly effective approach for developing more efficient, practical, and powerful reasoning LLMs. Figure 1 : An overview of DIET by mitigating LLM verbosity using difficulty-aware training. 2 Problem Formulation and Preliminaries 2.1 Token Efficiency in LLMs with Reasoning Capabilities Formally, we define the token efficiency problem studied in this paper as a multi-objective optimization challenge. Given a capable reasoning LLM policy \pi_{\theta} and a distribution of reasoning problems \mathcal{D} , we aim to find a policy \pi_{\theta}^{\prime} that optimizes both performance and token efficiency: \pi_{\theta}^{\prime}=\arg\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}[P(\pi,x)]-\alpha\cdot\mathbb{E}_{x\sim\mathcal{D}}[f(L(\pi,x))], (1) where L(\pi,x) represents the expected response length (in tokens) for prompt x under policy \pi , P(\pi,x) represents the performance (e.g., accuracy on math problems) on task x , \alpha>0 is a coefficient that controls the trade-off between performance and token efficiency, and f is a monotonically increasing transformation function with different implementations in prior work. The current paradigm of RL from outcome rewards typically focuses solely on maximizing performance without considering token efficiency ( \alpha=0 ), which leads to overthinking . Previous approaches to addressing this issue have generally attempted to incorporate token length into the reward function in an intuitive manner: r(x,y)=r_{\text{outcome}}(x,y)-\alpha\cdot f\left(L(y)\right), (2) where r_{\text{outcome}}(x,y) represents the outcome reward, L(y) is the length of response y in tokens. However, the uniform penalty \alpha treats all problems equally, failing to account for varying difficulty levels, and with inappropriate \alpha , it quickly leads to suboptimal tradeoffs, where performance degrades substantially as response length is reduced. 2.2 Model-Based Difficulty Estimation A critical insight of our work is that the optimal response length for a task should vary according to its difficulty. Intuitively, challenging problems may benefit from extended reasoning, while simpler questions can be answered concisely without sacrificing accuracy. We estimate the difficulty of a given problem based on the performance of the policy model itself during training. Formally, we define the estimated correctness for a given prompt x under policy \pi_{\theta} based on N sampled responses \{y_{i}\}_{i=1}^{N} : \hat{C}(x,\pi_{\theta})=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}(y_{i}\text{ is correct})\quad\text{where }y_{i}\sim\pi_{\theta}(\cdot|x). (3) The estimated difficulty \hat{D}(x,\pi_{\theta}) can then be defined as 1-\hat{C}(x,\pi_{\theta}) . This formulation captures an intuitive notion: tasks that the current policy consistently fails on ( \hat{C}\approx 0 ) are considered difficult, while those consistently answered correctly ( \hat{C}\approx 1 ) are considered easy. Notably, popular RL algorithms such

Read the complete text