Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy

Langzhou He, Junyou Zhu, Yue Zhou, Zhengyao Gu, Junhua Liu, Wei-Chieh Huang, Henry Peng Zou, David Wipf, Philip S. Yu, Qitian Wu, [0.5em] University of Illinois Chicago · 2026

Read Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy by Langzhou He, Junyou Zhu, Yue Zhou, Zhengyao Gu, Junhua Liu, Wei-Chieh Huang, Henry Peng Zou, David Wipf, Philip S. Yu, Qitian Wu, [0.5em] University of Illinois Chicago (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14558.txt.

Introduction

He, Zhu, Zhou, Gu, Liu, Huang, Zou, Wipf, Yu and Wu argue that standard policy-gradient training misallocates credit in agentic reinforcement learning, where long reasoning traces surround short actions. Their energy-based analysis finds training signal concentrates on action tokens, a pattern they call the action bottleneck. Their ActFocus reweighting downweights reasoning tokens and boosts uncertain action tokens, with claimed final-step gains over PPO and GRPO across four environments.

Catalog record

CREATORLangzhou He, Junyou Zhu, Yue Zhou, Zhengyao Gu, Junhua Liu, Wei-Chieh Huang, Henry Peng Zou, David Wipf, Philip S. Yu, Qitian Wu, [0.5em] University of Illinois Chicago
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64471
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.14558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy Langzhou He, Junyou Zhu, Yue Zhou, Zhengyao Gu, Junhua Liu, Wei-Chieh Huang, Henry Peng Zou, David Wipf, Philip S. Yu, Qitian Wu, [0.5em] University of Illinois Chicago Abstract Agentic reinforcement learning trains large language models using multi-turn trajectories that interleave long reasoning traces with short environment-facing actions. Common policy-gradient methods, such as PPO and GRPO, treat each token in a trajectory equally, leading to uniform credit assignment. In this paper, we critically demonstrate that such uniform credit assignment largely misallocates token-level training signals. From an energy-based modeling perspective, we show that token-level training signals, quantified by their correlations with reward variance of different rollouts sampled from a given prompt, concentrate sharply on action tokens rather than reasoning tokens, even though action tokens account for only a small fraction of the trajectory. We refer to this phenomenon as the Action Bottleneck. Motivated by this observation, we propose an embarrassingly simple token reweighting approach, ActFocus , that downweights gradients on reasoning tokens, along with an additional energy-based redistribution mechanism that further increases the weights on action tokens with higher uncertainty. Across four environments and different model sizes, ActFocus consistently outperforms PPO and GRPO, yielding final-step gains of up to 65.2 and 63.7 percentage points, respectively, without any additional runtime or memory cost. 1 Introduction As large language models (LLMs) are increasingly deployed as autonomous agents that combine reasoning [ 8 , 9 ] , planning [ 37 ] , and tool use [ 31 ] to solve multi-turn, long-horizon tasks, reinforcement learning (RL) has emerged as the natural post-training framework for sequential decision-making settings [ 29 , 27 , 32 , 1 ] . Despite rapid progress with different policy optimization methods [ 20 , 22 ] , training LLM agents with RL remains difficult. One of the core challenges is credit assignment [ 17 , 10 ] , where, in multi-turn trajectories, a sparse episodic reward must be propagated back through thousands of generated tokens, making it difficult to determine which parts of a trajectory should drive learning. The problem is compounded by optimization instability [ 5 ] , as delayed feedback, long generated traces, and noisy policy updates often lead to degraded late-stage performance or collapse. Recent evidence shows that multi-turn, underspecified conversations substantially increase outcome variability for LLMs, sharpening the need for reliable credit assignment in agentic RL [ 11 ] . In a typical agentic RL rollout, such outcome variability often hinges on a few critical environment-facing action tokens that drive state transitions, yet these tokens are buried within massive amounts of reasoning tokens in the trajectory [ 35 , 19 , 30 ] . Even though these action tokens are the only parts of the output that directly interact with the environment, standard policy optimization methods treat all generated tokens uniformly [ 20 , 14 ] . Consequently, gradient updates are spread equally across the whole trajectory, diluting training signals for the critical action tokens that trigger the decision-making [ 2 , 26 ] . In multi-turn settings, this mismatch compounds with each additional turn: reasoning tokens keep accumulating while the environment is driven by only a few informative actions. Figure 1 shows that this structural mismatch manifests as token-level signal sparsity in agentic RL. In a representative Sokoban trajectory, action tokens constitute only 4% of the generated output, while reasoning tokens account for the remaining 96%. Under uniform credit assignment, gradient mass therefore follows token count and is dominated by the reasoning majority. However, in this paper, we demonstrate that the training signals follow the opposite pattern. We quantify each token’s training signal as its predictive uncertainty under a frozen reference model, measured by the free energy of its next-token predictive distribution. 1 1 1 An ideal indicator for quantifying training signals should reveal where outcome uncertainty concentrates, stay stable across training, and be computationally inexpensive. Energy, used in this work, satisfies all of these criteria; alternatives such as entropy and log-likelihood are compared empirically in Sec. 5.6 . To further evaluate how informative the signal is for training, we hinge it on outcome variability: for each prompt, we calculate the variance of episodic rewards \sigma_{g} across multiple rollouts (sampled from the same prompt), which captures how uncertain the current policy is about the prompt’s outcome. We find that the mean energy of action tokens is strongly correlated with reward variance (with \rho=+0.537 ), while the mean energy of reasoning tokens or full trajectories, which average over both action and reasoning tokens, is nearly uninformative (with \rho\approx 0 ). This suggests that informative training signals are concentrated in a small number of action tokens rather than being distributed uniformly across the trajectory. We refer to this phenomenon as the Action Bottleneck . Figure 1 : Action Bottleneck in agentic reinforcement learning. Left: A real response from a trajectory, where reasoning tokens far outnumber action tokens. Middle: Action tokens constitute only 4% of model-generated tokens; our token-level reweighting redirects gradient mass towards them. Right: Training signal concentrates in action spans. Results are from Sokoban 3B. Motivated by this observation, we propose an energy-based approach to token-level credit assignment in agentic RL. The method is conceptually intuitive and can be decomposed into two aspects: 1) We downweight gradients on reasoning tokens, which is equivalent to increasing the weights on action tokens and helps to redirect gradient mass toward the sparse action span that determines environmental outcomes; 2) We utilize token-level energy to further redistribute the weights across action tokens, which prioritizes action tokens with high energy (i.e., more predictive uncertainty). In this way, we transform our empirical observations into an actionable approach that systematically guides the model to better leverage the informative training signals underlying a few critical tokens in the long trajectory. We evaluate our approach, ActFocus , on four representative multi-turn agentic environments using Qwen2.5-Instruct [ 18 ] models at multiple scales, and train each configuration with both PPO [ 20 ] and GRPO [ 22 ] . Through these experiments, we find that ActFocus produces consistent performance gains over PPO and GRPO, both of which use uniform credit assignment, with final success-rate improvements of up to 65.2 and 63.7 percentage points, respectively. Beyond absolute performance metrics, we observe that ActFocus effectively improves GRPO’s training stability by substantially reducing peak-to-final degradation. Our main contributions are summarized as follows: (i) We analyze token-level attributions in agentic RL training through an energy-based modeling perspective and reveal a counter-intuitive phenomenon: while reasoning tokens dominate in long-horizon multi-turn trajectories, infrequent action tokens contribute the majority of informative training signals. In particular, the uncertainty in action tokens, as measured by the free energy of next-token predictive distributions, is highly correlated with outcome variability. (ii) We propose a token reweighting mechanism that downweights gradients on reasoning tokens and additionally redistributes the weights on action tokens according to their uncertainty levels as characterized by the free energy. This simple scheme shifts gradient mass towards the most informative action tokens while remaining fully compatible with common policy-gradient algorithms. (iii) Through extensive experiments across four environments and multiple model scales, ActFocus consistently outperforms PPO and GRPO, with average gains of 17.0 and 34.6 percentage points

Read the complete text