AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning

Haotian Zhao, Songlin Zhou, Yuxin Zhang, Stephen S.-T. Yau, Wenyu Zhang, Lun Tian, Tianshu Zhu, Yifeng Huang, Yucheng Zeng, Jingnan Gu, Daxiang Dong, Jianmin Wu · 2026

Read AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning by Haotian Zhao, Songlin Zhou, Yuxin Zhang, Stephen S.-T. Yau, Wenyu Zhang, Lun Tian, Tianshu Zhu, Yifeng Huang, Yucheng Zeng, Jingnan Gu, Daxiang Dong, Jianmin Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00425.txt.

Introduction

Haotian Zhao and 11 co-authors improve how language model agents earn credit for good steps in long tasks. Reinforcement learning has sharpened multi-turn agents, but rewards usually arrive only at the end, leaving each step's contribution unclear. Dense step-by-step supervision helps yet adds tuning burden and can limit generalization. Their method, AEM, needs no extra supervision at all.

AEM works by modulating entropy, the model's uncertainty, at the level of whole responses rather than single tokens, matching how an agent's action actually affects its environment. Theory in the paper ties entropy drift to the balance between a response's advantage and its surprisingness. AEM turns that into an uncertainty proxy that rescales advantages, sliding the balance from exploration toward exploitation as training proceeds. Across web shopping, household, and software benchmarks, it consistently improved strong baselines.

Catalog record

CREATORHaotian Zhao, Songlin Zhou, Yuxin Zhang, Stephen S.-T. Yau, Wenyu Zhang, Lun Tian, Tianshu Zhu, Yifeng Huang, Yucheng Zeng, Jingnan Gu, Daxiang Dong, Jianmin Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00425

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning Haotian Zhao, Songlin Zhou, Yuxin Zhang, Stephen S.-T. Yau, Wenyu Zhang, Lun Tian, Tianshu Zhu, Yifeng Huang, Yucheng Zeng, Jingnan Gu, Daxiang Dong, Jianmin Wu Abstract Reinforcement learning (RL) has substantially improved the ability of large language model (LLM) agents to interact with environments and solve multi-turn tasks. However, effective agentic RL remains challenging: sparse outcome-only rewards provide limited guidance for assigning credit to individual steps within long interaction trajectories. Existing approaches often introduce dense intermediate supervision, such as process reward models or auxiliary self-supervised signals, which increases supervision and tuning complexity and may limit generalization across tasks and domains. We present AEM, a supervision-free credit assignment method that adaptively modulates entropy dynamics during RL training to improve the exploration-exploitation trade-off. Since in agentic RL the environment is typically affected by a complete response, rather than an individual token, our analysis lifts entropy dynamics from the token level to the response level, aligning uncertainty estimation with the effective action granularity of LLM agents and reducing sensitivity to token-level sampling noise. We further show that entropy drift under natural-gradient updates is governed by the interaction between the sampled-response advantage and its relative surprisal. Motivated by this result, AEM derives a practical response-level uncertainty proxy and uses it to rescale advantages, leveraging the evolving balance between positive and negative samples to naturally transition from exploration to exploitation. Extensive experiments on ALFWorld, WebShop, and SWE-bench-Verified with models ranging from 1.5B to 32B demonstrate that AEM consistently improves strong RL baselines, including a +1.4% gain when integrated into a state-of-the-art software-engineering RL training framework. 1 Introduction Large language models (LLMs) are increasingly being deployed as interactive agents that solve complex tasks through multi-turn reasoning ( 37 ; 46 ) , tool use ( 29 ; 36 ) , and sustained interaction with external environments ( 4 ; 11 ) . In such agentic settings, LLMs are no longer evaluated solely by isolated generation quality, but by their ability to make sequential decisions ( 47 ) : repeatedly observing environment feedback, selecting actions, and refining their behavior across long interaction trajectories ( 30 ; 10 ) . This shift has enabled rapid progress in challenging domains such as autonomous software engineering ( 39 ; 40 ) , embodied assistance ( 41 ; 17 ) , and GUI navigation ( 45 ; 18 ) . Reinforcement learning (RL) has emerged as a central paradigm for improving such agents ( 9 ) , with group-based methods such as GRPO ( 27 ) providing an effective value-free alternative to actor-critic training 16 ; 21 . However, extending these methods from single-turn post-training to multi-turn agentic RL remains fundamentally challenging. Under such settings, feedback is sparse and outcome-based: the agent receives a reward only after completing a long trajectory ( 12 ) . As a result, different steps within the same trajectory often receive nearly indistinguishable learning signals, leading to ambiguous credit assignment and inefficient policy improvement. Existing approaches address this issue by introducing denser credit signals. Reward shaping-based methods , such as process reward models ( 19 ) , provide dense step-level supervision but require additional models or annotations; tree-structured optimization methods , such as Tree-GRPO ( 7 ) and ATPO ( 3 ) , enable fine-grained credit propagation via branching trajectories but incur high computational overhead in multi-turn settings; self-supervised methods (such as GiGPO ( 12 ) and IGPO ( 33 ) ) infer step-level signals from trajectory structure without auxiliary supervision but are prone to context inconsistency, grouping bias, and heavy dependence on structural assumptions, which limit robustness and generalization. Collectively, these limitations call for a scalable, fine-grained credit assignment framework that does not rely on extra supervision, heavy computation, and restrictive structural assumptions. Specifically, we notice that: (i) the policy’s own entropy already provides an intrinsic signal for credit assignment: high-entropy responses typically reflect exploratory decisions, whereas low-entropy responses indicate more confident policy behavior; (ii) each completed response 1 1 1 In practice, a response usually combines reasoning and acting; in RL theory, it’s the ”action” sampled from the policy. To avoid ambiguity, we use the term ”response”. is the effective unit that changes the environment state. Therefore, we treat response-level entropy as an intrinsic signal for credit modulation. We demonstrate that the entropy drift induced by a sampled response is governed by the interaction between its advantage and relative response surprisal. This motivates A daptive E ntropy M odulation (AEM), a credit assignment algorithm that uses a practical response-entropy proxy to rescale response-level advantages. AEM adaptively preserves exploration early in training and promotes exploitation as successful samples become more prevalent, enhancing response diversity early in training while enabling more complete convergence in later stages. Our contributions are three-fold. • We provide a response-level theoretical analysis of entropy dynamics in multi-turn agentic RL. By showing that entropy drift is determined by the interaction between sampled-response advantage and relative surprisal, our analysis reveals response-level uncertainty as a principled intrinsic signal for credit assignment. • We propose AEM, a supervision-free, lightweight, plug-in method that modulates response-level advantages using an entropy-derived uncertainty proxy. By leveraging the evolving balance between positive and negative samples during training, AEM adaptively guides the policy from early-stage exploration to late-stage exploitation. • We conduct extensive experiments on ALFWorld, WebShop, and SWE-bench-Verified using models from 1.5B to 32B. AEM consistently improves multiple strong group-based RL baselines, with peak gains of 8.8% on GRPO with Qwen2.5-1.5B on ALFWorld, and a +1.4% improvement when applied to DeepSWE on SWE-bench-Verified, demonstrating the effectiveness and generality of entropy-aware response-level credit modulation. 2 Related Work From LLMs to Agentic RL. Representative works such as ReAct ( 43 ) and Toolformer ( 25 ) demonstrate that LLMs can interleave reasoning with actions and external tool calls, shifting the role of LLMs from passive generators to interactive decision-makers. Training such agents increasingly relies on RL, where group-based methods such as RLOO ( 1 ) and GRPO ( 27 ) have emerged as a dominant approach. Extending these methods from single-turn to multi-turn agentic settings exacerbates sparse rewards: feedback arrives only at the end, providing little guidance for intermediate decisions. The lack of step-level supervision yields high-variance gradients and ambiguous credit assignment, obscuring which intermediate actions should be reinforced or discouraged. Credit Assignment in Agentic RL. Credit assignment is a long-standing challenge in agentic RL with delayed and sparse rewards. Existing efforts for step-level credit assignment in agentic RL differ mainly in where and how credit signals are derived. Some rely on external signals , such as value functions or step-level supervision ( 26 ; 19 ) , but introduce additional modeling and scaling overhead. Others derive credit internally from sampled trajectories ( 12 ; 33 ) , avoiding auxiliary supervision; some methods infer credit implicitly from trajectory attributes, while others further refine credit through structured propagation ( 3 ; 7 ) or reward redistribution ( 34 ) , improving credit granularity but often incurring additional computational cost in multi-turn settings. To address these limitations, a more general, lightweight, and adaptive credit assignment method is needed. Entropy-Aware Policy Optimization. Entropy has long been used in RL as a regularization term for promoting exploration

Read the complete text