IAPO : Information-Aware Policy Optimization for Token-Efficient Reasoning

Yinhan He, Yaochen Zhu, Mingjia Shi, Wendy Zheng, Lin Su, Xiaoqing Wang, Qi Guo, Jundong Li · 2026

Read IAPO : Information-Aware Policy Optimization for Token-Efficient Reasoning by Yinhan He, Yaochen Zhu, Mingjia Shi, Wendy Zheng, Lin Su, Xiaoqing Wang, Qi Guo, Jundong Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19049.txt.

Introduction

Long chains of thought raise accuracy, but they cost dearly at inference time. Yinhan He, with seven co-authors, rethinks token-efficient post-training in 2026. Existing reward-shaping methods control effort only at sequence level. They cannot steer effort token by token.

The authors propose IAPO, an information-theoretic frame. It assigns each token an advantage from its conditional mutual information with the final answer. Informative steps gain weight. Low-utility exploration gets suppressed. Theory shows verbosity can fall monotonically without harming correctness. Tests show accuracy rises while reasoning length falls by over a third. IAPO beats existing token-efficient methods across reasoning datasets. Information-aware shaping thus offers a general path to cheaper reasoning.

Catalog record

CREATORYinhan He, Yaochen Zhu, Mingjia Shi, Wendy Zheng, Lin Su, Xiaoqing Wang, Qi Guo, Jundong Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE100 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-19049

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IAPO : Information-Aware Policy Optimization for Token-Efficient Reasoning Yinhan He, Yaochen Zhu, Mingjia Shi, Wendy Zheng, Lin Su, Xiaoqing Wang, Qi Guo, Jundong Li Abstract Large language models increasingly rely on long chains of thought to improve accuracy, yet such gains come with substantial inference-time costs. We revisit token-efficient post-training and argue that existing sequence-level reward-shaping methods offer limited control over how reasoning effort is allocated across tokens. To bridge the gap, we propose IAPO , an information-theoretic post-training framework that assigns token-wise advantages based on each token’s conditional mutual information (MI) with the final answer. This yields an explicit, principled mechanism for identifying informative reasoning steps and suppressing low-utility exploration. We provide a theoretical analysis showing that our IAPO can induce monotonic reductions in reasoning verbosity without harming correctness. Empirically, IAPO consistently improves reasoning accuracy while reducing reasoning length by up to 36%, outperforming existing token-efficient RL methods across various reasoning datasets. Extensive empirical evaluations demonstrate that information-aware advantage shaping is a powerful and general direction for token-efficient post-training. The code is available at https://github.com/YinhanHe123/IAPO . † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : Reasoning verbosity of RL post-trained LLMs. (a) Comparison of reasoning length between LLM (DeepSeekR1-Distilled-Qwen-1.5B ( 10 ) ) and a human volunteer on math problems ( 17 ) . (b) Illustration of why the reasoning generated by the LLM are unnecessarily verbose. Reinforcement learning (RL)-based post-training has become one of the most effective tools for strengthening the reasoning capabilities of large language models (LLMs) ( 12 ) . Recently, group relative policy optimization (GRPO) ( 25 ) and its variants have proven especially powerful: by contrasting multiple sampled solutions and explicitly rewarding reasoning chains that lead to better answers, these methods incentivize deeper, more structured multi-step reasoning, thus delivering substantial gains across diverse reasoning benchmarks. However, GRPO also comes with an often-overlooked cost. Post-trained LLMs tend to produce more reasoning than necessary. While additional reasoning steps enable more computational power of the base LLM, the generated content could be redundant, circular, or uninformative. As shown in Fig. 1 , DeepSeekR1-Distilled-Qwen-1.5B ( 10 ) and a Ph.D.-level volunteer both achieve perfect accuracy on 20 MATH-500 problems ( 17 ) , yet the model generates substantially longer reasoning (1,658 vs. 264 tokens on average). Such verbosity inflates inference latency and computational cost, which scale quadratically with sequence length, while offering diminishing returns in correctness. This mismatch motivates the central question: Can we retain the reasoning strength achieved by RL post-training while reducing unnecessary reasoning tokens? Several recent approaches have attempted to answer this question by shaping advantages in GRPO ( 25 ) to favor brevity ( 31 ; 27 ; 14 ; 40 ) . However, the semantics of the generated reasoning are overall overlooked. Generally, existing methods fall into two broad categories. Length-based methods ( 27 ; 40 ) favor shorter outputs by assigning uniformly higher advantages to all tokens in concise outputs, regardless of whether they are actually informative. Position-based methods ( 9 ; 14 ; 40 ) penalize later tokens via position-dependent advantage decay, even if some later tokens can be crucial for producing correct answers. Both paradigms share a fundamental limitation: they are content-agnostic , i.e., they determine a token’s advantage by completion length or token position, instead of the information it contributes to the correctness of the final answer. Consequently, current approaches struggle to distinguish essential reasoning from redundant tokens. To overcome this limitation, we propose I nformation- A ware P olicy O ptimization ( IAPO ), a novel post-training framework that brings information-theoretic awareness into intra-completion token-level advantages for policy optimization. Specifically, IAPO consists of two modules. The information-aware advantage shaping module assigns token-level advantages by quantifying each token’s contribution to answer correctness through its conditional mutual information (MI) with the final answer, conditioned on the preceding partial completion. This module provides a principled mechanism for identifying informative reasoning steps and reducing low-utility generation. The efficient conditional MI estimation module efficiently estimates token-level conditional MI values utilizing an early-exit–based conditional MI estimator, together with KV-cache preloading and chunk-wise forwarding techniques, to significantly reduce the computational overhead of MI estimation, making our proposed IAPO computationally tractable at scale. Empirically, IAPO delivers substantial gains in token efficiency while preserving strong reasoning performance. Applied to Qwen2.5-7B-Instruct ( 38 ; 32 ) , IAPO achieves a 36% reduction in reasoning length relative to the base model and a significant improvement over the best existing token-efficient RL baselines on GSM8K ( 7 ) , without sacrificing accuracy. Our contributions are as follows. (1) Problem Formulation. We formalize token-efficient post-training as maximizing the ratio of expected task accuracy to completion length, capturing the objective of achieving high reasoning performance with minimal token consumption. (2) Theoretical Framework. We propose IAPO , a token-level advantage-shaping framework that assigns advantages based on conditional MI between tokens and final answers. We provide a theoretical analysis demonstrating how IAPO reduces the expected completion length while maintaining model accuracy. (3) Efficient Module Design. We introduce an early-exit conditional MI estimator along with KV-cache preloading and chunk-wise forwarding techniques, making token-level MI computation tractable for modern LLMs at scale. (4) Extensive Empirical Validation. Across multiple reasoning datasets and model scales, IAPO consistently achieves state-of-the-art token efficiency compared with existing token-efficient RL post-training baselines for LLMs. 2 Preliminaries and Problem Definition Preliminaries. Let q\sim Q denote an input query (e.g., a math problem), and let o be a completion sampled from an LLM policy \pi_{\theta} . Post-training aims to optimize a \pi_{\theta} which maximize the expected rewards r (a value quantifying the quality of o ), i.e., \max_{\theta}{\mathbb{E}_{q\sim Q,o\sim\pi_{\theta}(o|q)}[r(o)]} . For each query q , GRPO ( 25 ) samples a group of G completions \{o_{i}\}_{i=1}^{G} from a frozen policy \pi_{\theta_{\mathrm{old}}} . Each completion o_{i} receives a reward r_{i}\in\mathbb{R} from a reward model. GRPO computes advantages as group-wise normalized rewards \tilde{A}_{i}=\frac{r_{i}-\mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})}, where \mathbf{r}=\{r_{i}\}_{i=1}^{G} . The normalized advantage is then uniformly assigned to all token positions t s in o_{i} as token-wise advantages \tilde{A}_{i,t}=\tilde{A}_{i},\forall t. The GRPO objective maximizes a clipped policy-gradient surrogate ( 11 ) with KL regularization: \displaystyle J_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{q,\{o_{i}\}}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\min\Big(\rho_{i,t}(\theta)\,\tilde{A}_{i,t}, \displaystyle\mathrm{clip}(\rho_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,\tilde{A}_{i,t}\Big)-\beta\,D_{\mathrm{KL}}(\pi_{\theta}\,\|\,\pi_{\mathrm{ref}})\Bigg], where \rho_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})}. While GRPO ( 25 ) effectively improves reasoning accuracy, its uniform token-level advantage fails to distinguish informative tokens from redundant ones, often leading to verbose completions. Problem 1 (Token-Efficient Post-Training) . Given a query–answer distribution (Q,Y) , we aim to learn a policy \pi_{\theta} that maximizes reasoning accuracy per generated token for samples (q,y) drawn from (Q,Y) : \displaystyle\max_{\theta} \displaystyle\frac{\mathbb{E}_{q\sim Q,\,o\sim\pi_{\theta}}\left[\mathbb{I}\{o\mathrm{~is~correct}\}\right]}{\mathbb{E}_{q\sim Q,\,o\sim\pi_{\theta}}\left[|o|\right]} (1) \displaystyle\text{s.t.} \displaystyle\mathbb{E}_{q\sim Q,\,o\sim\pi_{\theta}}\left[\mathbb{I}\{o\mathrm{~is~correct}\}\right]\geq\tau, Here, \tau denotes a minimum effectiveness threshold that prevents degenerated policies, e.g., trivially short but incorrect completions, from being mistakenly regarded as token-efficient. This formulation captures the central objective of token-efficient reasoning: preserving task-relevant information while eliminating redundant token generation. 3 Proposed Methodology Figure 2 : Illustration of the information-aware advantage shaping module, where s_{i,j} and c_{i,j} are token-wise advantages of the informativeness level and exploration adjustment of the j th token in the i th completion o_{i} of the completions group \{o_{i}\}_{i=1}^{G} . We first present an overview of the IAPO framework, followed by an illustration of its two critical modules: the information-aware advantage shaping module and the efficient conditional MI

Read the complete text