InfoPO: Information-Driven Policy Optimization for User-Centric Agents

Fanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu, Yuyu Luo, Bang Liu · 2026

Read InfoPO: Information-Driven Policy Optimization for User-Centric Agents by Fanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu, Yuyu Luo, Bang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00656.txt.

Introduction

Kong and five colleagues sharpen multi turn agent learning with InfoPO, which treats interaction as active uncertainty reduction. Current methods score whole trajectories, blurring which turns deserve credit and weakening advantage signals within rollout groups. InfoPO rewards turns whose feedback measurably shifts the agent's next action distribution against a masked feedback counterfactual, then fuses that signal with task outcomes through adaptive variance gating. Across intent clarification, collaborative coding, and tool aided decisions, InfoPO beats prompting and multi turn reinforcement learning baselines and holds up under simulator shifts.

Catalog record

CREATORFanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu, Yuyu Luo, Bang Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78219
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.00656

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InfoPO: Information-Driven Policy Optimization for User-Centric Agents Fanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu, Yuyu Luo, Bang Liu Abstract Real-world user requests to LLM agents are often underspecified. Agents must interact to acquire missing information and make correct downstream decisions. However, current multi-turn GRPO-based methods often rely on trajectory-level reward computation, which leads to credit assignment problems and insufficient advantage signals within rollout groups. A feasible approach is to identify valuable interaction turns at a fine granularity to drive more targeted learning. To address this, we introduce InfoPO (Information-Driven Policy Optimization), which frames multi-turn interaction as a process of active uncertainty reduction and computes an information-gain reward that credits turns whose feedback measurably changes the agent’s subsequent action distribution compared to a masked-feedback counterfactual. It then combines this signal with task outcomes via an adaptive variance-gated fusion to identify information importance while maintaining task-oriented goal direction. Across diverse tasks, including intent clarification, collaborative coding, and tool-augmented decision making, InfoPO consistently outperforms prompting and multi-turn RL baselines, exceeding GRPO-based methods by 14% to 16%. It also demonstrates robustness under user simulator shifts and generalizes effectively to environment interactive tasks. Overall, InfoPO provides a principled and scalable mechanism for optimizing complex agent user collaboration. Code is available at https://github.com/kfq20/InfoPO . 1 Introduction The rapid evolution of Large Language Models (LLMs) has enabled interactive agents that assist users in complex, multi-turn tasks ( Zhu et al. 2025 ; Li et al. 2025a ; Liu et al. 2025a ; Li et al. 2025e ) . In the user-centric setting where agents must serve human users with underspecified goals, they must bridge a fundamental gap between often ambiguous human intentions and the precise parameters required for machine execution ( Norman 1986 ; Clark & Brennan 1991 ; Budzianowski et al. 2018 ) . For example, a request like “book me a flight next week” is not directly actionable until the agent elicits missing constraints such as dates, departure airport, budget, and flexibility. Therefore, an effective interaction should both increase the agent’s knowledge about the user’s true intent and advance the task toward completion. Mastering this interplay between intent elicitation and task execution remains a core challenge for building autonomous agents that operate reliably under partial information ( Deng et al. 2025 ; Qian et al. 2025b ; He et al. 2025 ; Luo et al. 2018 ) . Figure 1 : Standard GRPO vs. InfoPO. Standard GRPO yields zero reward for ”honorable failures” (correct elicitation, failed execution). InfoPO solves this via counterfactual masking to provide dense, turn-level information-gain rewards. This challenge calls for a principled approach to learning an interaction policy. Reinforcement learning (RL) stands out as the standard paradigm for this purpose, as it enables an agent to autonomously discover effective strategies from feedback in sequential decision-making settings ( Zhang et al. 2025 ; Liu et al. 2025b ) . However, applying RL to multi-turn agentic tasks exposes its Achilles’ heel: the long-horizon credit assignment problem. Rewards are often sparse and delayed until task completion, making it difficult to attribute outcomes to intermediate decisions ( Zhou et al. 2025 ; Wei et al. 2025 ) . This issue is particularly pronounced under GRPO-based methods ( Shao et al. 2024 ; Yu et al. 2025 ) , where policy updates rely on reward variation within a rollout group. Moreover, many recent works on multi-turn RL aggregate terminal and intermediate signals into a single trajectory-level score for advantage estimation, which limits fine-grained supervision across the interaction ( Qian et al. 2025a ; Qian et al. 2025c ; Wang et al. 2025 ; Jin et al. 2025 ; Lin et al. 2025 ) . In user-centric environments, such granularity matters even more. A small number of clarification decisions can determine feasibility and downstream success, and RL training frequently relies on LLM-simulated users ( Zhao et al. 2025b ; Cai et al. 2025 ; Li et al. 2025d ; Li et al. 2025c ) , making sample efficiency also a critical constraint. To address the above limitation, we propose Information-Driven Policy Optimization ( InfoPO ), which treats multi-turn interaction as a process of active uncertainty reduction (as illustrated in Figure 1 ). InfoPO defines a turn-level counterfactual information-gain reward that credits the action for the information it elicits. After the agent acts, the user or environment provides feedback. We then ask how this feedback would change the agent’s next decision. Concretely, we score the same collected trajectory under two conditions: (i) the factual history that includes the feedback and (ii) a counterfactual history where the feedback is replaced by a masked placeholder. We compare the policy’s probability assigned to the actual next action under the two conditions. The resulting distribution gap is attributed to the action that triggered the feedback, rewarding turns that meaningfully reshapes downstream choices. This process does not need additional interactions with environments. To keep this intrinsic signal aligned with task completion, InfoPO uses an adaptive variance-gated fusion to combine information gain with outcome-based updates. When outcome rewards within a rollout group are non-discriminative, group-relative advantages can be near zero, leading to learning stagnation. In this case, the gate increases the weight on information gain to maintain a usable training signal. As outcomes become more discriminative, the gate shifts weight back to the task objective to support eventual success. We also provide an information-theoretic interpretation of the learning signal. In expectation, the per-turn information gain reward corresponds to a conditional mutual information between feedback and the agent’s next action, while its cumulative form represents the directed information flow from observations to decisions. Crucially, we prove that a minimum cumulative information gain is a strictly necessary resource for achieving task success, providing a theoretical lower bound that links uncertainty reduction directly to the probability of reaching the goal. We evaluate InfoPO on three representative interactive benchmarks, UserGym ( Qian et al. 2025c ) , ColBench ( Zhou et al. 2025 ) , and the long-horizon \tau^{2} -Bench ( Barres et al. 2025 ) , which together span intent clarification and preference elicitation, collaborative code generation, and tool-augmented decision making. InfoPO consistently improves task performance and learning stability over strong prompting and RL baselines. To attribute these gains, we run component ablations and diagnostic analyses, uncovering a learned interaction pattern that resolves intent uncertainty early before downstream commitments. We further test robustness beyond user-centric benchmarks by applying InfoPO to non-user-interactive tasks and by swapping the simulated user at evaluation time, indicating that InfoPO learns broadly useful interaction strategies. In summary, our contributions are threefold: (1) we introduce InfoPO, an information-driven RL method for multi-turn interaction that provides dense turn-level credit to enable effective learning under sparse or delayed outcomes; (2) we provide an information-theoretic grounding that links the proposed signal to conditional mutual information and directed information, clarifying the role of information flow from feedback to actions in learning; (3) we deliver a comprehensive empirical study with ablations and diagnostic analyzes across multiple interactive benchmarks, showing improved task performance and training stability, and generalization to variant users and environments. 2 Related Works User-centric agents. User-centric agents move beyond “task completion” toward inferring latent intent, preferences, and user state under multi-turn interaction. On the intent/need side, recent

Read the complete text