Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs

Siyu Zhu 1 * † Yanbin Jiang 1 * Hejian Sang 1 * Shao Tang 1 * Qingquan Song 1 * Biao He 1 Rohit Jain 1 Zhipeng Wang 1 Alborz Geramifard 1 * † 1 LinkedIn Corporation, CA, USA · 2025

Read Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs by Siyu Zhu 1 * † Yanbin Jiang 1 * Hejian Sang 1 * Shao Tang 1 * Qingquan Song 1 * Biao He 1 Rohit Jain 1 Zhipeng Wang 1 Alborz Geramifard 1 * † 1 LinkedIn Corporation, CA, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25779.txt.

Introduction

Siyu Zhu and co-authors train travel-planning agents with shaped rewards in a 2025 paper. Their system, Planner-R1, learns by reinforcement on the TravelPlanner benchmark. It reached a high final-pass rate with only a small set of training queries, beating a strong general model baseline.

Small models responded best to dense, step-by-step reward signals. An eight-billion-parameter model matched larger rivals while using far less compute and memory. Big models were steadier under sparse rewards but gained less from shaping and varied more between runs. Curriculum learning added no clear gain. The lesson is practical: reward design can matter more than model size for agent work.

Catalog record

CREATORSiyu Zhu 1 * † Yanbin Jiang 1 * Hejian Sang 1 * Shao Tang 1 * Qingquan Song 1 * Biao He 1 Rohit Jain 1 Zhipeng Wang 1 Alborz Geramifard 1 * † 1 LinkedIn Corporation, CA, USA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25779

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs Siyu Zhu 1 * † Yanbin Jiang 1 * Hejian Sang 1 * Shao Tang 1 * Qingquan Song 1 * Biao He 1 Rohit Jain 1 Zhipeng Wang 1 Alborz Geramifard 1 * † 1 LinkedIn Corporation, CA, USA Abstract We investigated Agentic RL with large language models on the TravelPlanner benchmark. Our approach, Planner-R1 , achieved a 56.9% final-pass rate with only 180 training queries, a 2.7 × 2.7\times improvement over GPT-5’s 21.2 % 21.2\% baseline and the strongest agentic result on the public leaderboard. A central finding was that smaller models (8B) were highly responsive to reward shaping: with dense process-level signals, they reached competitive performance while being 3.5 × 3.5\times more compute-efficient and 1.5 × 1.5\times more memory-efficient than 32B models. Larger models were more robust under sparse rewards but exhibited smaller relative gains from shaping and higher variance across runs. While curriculum learning offered no significant benefit, shaped rewards consistently amplified learning dynamics, making 8B models the most efficient setting for agentic RL. Crucially, these gains did not come at the cost of overfitting: fine-tuned models mostly maintained or exceeded baseline performance on out-of-domain tasks, including Multi-IF , NaturalPlan , and τ \tau - Bench . These results establish reward shaping as a decisive lever for scaling agentic RL, highlight the competitive strength of smaller models, and demonstrate that efficiency can be achieved without sacrificing generalization. 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding author: Siyu Zhu < [email protected] >, Alborz Geramifard < [email protected] >. Figure 1: Final-pass rate on the leaderboard test set for tool-use travel planning. Our Planner-R1 models outperformed SOTA LLMs reaching 56.9\% average final pass rate. 1 Introduction Large Language Models (LLMs) have recently posted striking gains in deliberate reasoning and decision making, propelled in part by large-scale reinforcement learning (RL) that trains models to think before they answer OpenAI et al. 2024a ; Guo et al. 2025 . Beyond language understanding, LLM agents now demonstrate emerging competence in structured reasoning, tool use, and multi-step problem solving across embodied and web environments Wang et al. 2023a ; Huang et al. 2024 ; Feng et al. 2025 . Yet turning these abilities into reliable long-horizon execution under real-world constraints remains challenging: prompting-only agents such as ReAct and Reflexion frequently mis-sequence actions, loop, or hallucinate when tasks demand coordinated tool use and strict constraint satisfaction Yao et al. 2023b ; Shinn et al. 2023 . Planning tasks such as meeting scheduling and multi-day itineraries are demanding: agents must coordinate heterogeneous tools (calendars, maps, flights, booking APIs), satisfy hard, interdependent constraints , and maintain global consistency over long horizons. TravelPlanner makes these difficulties concrete by casting travel itinerary creation as tool-augmented, constraint-driven planning Xie et al. 2024 . The benchmark provides a sandbox with nearly four million records and 1{,}225 curated intents with reference plans, and evaluates whether an agent can gather evidence via tools and synthesize itineraries that satisfy both explicit user constraints and commonsense feasibility. At release, even strong models struggled—e.g., GPT-4-Turbo with ReAct achieved only a 0.6\% final pass rate on the 1{,}000 -example test split—underscoring the gap between fluent language modeling and dependable constraint-aware planning Xie et al. 2024 . To close this gap, researchers have explored different training paradigms. A natural starting point is behavior cloning via supervised fine-tuning (SFT), where a teacher generates “golden” trajectories and a policy maximizes their likelihood, often masking environment observations and tool outputs. While simple and widely used, SFT largely imitates expert behavior and is brittle under distribution shift or suboptimal data. This motivates the search for approaches that directly optimize for end-task success rather than imitation fidelity. RL provides precisely such a mechanism: rewards encode task success, and the policy is updated to increase the likelihood of action sequences that satisfy constraints while suppressing those that fail. Recent work has shown that RL can deliver state-of-the-art gains in model-based reasoning and planning OpenAI et al. 2024a ; Guo et al. 2025 , making it a promising direction for tackling long-horizon tool use in TravelPlanner . In addition to model performance, there is growing interest in building efficient agentic systems with smaller models Belcak et al. 2025 . Such models show promising potential for inference and training efficiency, but there remains limited understanding of how agentic RL can best improve their performance without overfitting. Our study addresses this gap by examining how model size, reward shaping, and efficiency interact in agentic RL. We formulate TravelPlanner as a multi-step, tool-use MDP with constraint-aware planning, where the agent gathers missing facts, reconciles conflicts, and outputs a structured itinerary. Training uses agentic RL with trajectory-level rewards gated by schema validity. Our main focus is the role of reward density : we vary feedback from dense, process-level signals to sparse final-pass rewards, and also test a curriculum that transitions between them. All reward variants are properly shaped , ensuring they converge to the same optimal policy while revealing how granularity influences learning dynamics. Our contributions are summarized below. • SOTA Tool-Use on TravelPlanner Planner-R1-32B achieved a \mathbf{56.9}\% final-pass rate on the official 1,000-query test split, a 2.7\times improvement over GPT-5. This is the strongest agentic result on TravelPlanner , demonstrating that RL-tuned models can surpass state-of-the-art proprietary models. * * * Hao et al. 2025 achieved 93.9% correctness with external SAT/SMT solvers; our focus is on end-to-end agentic planning without such solvers. • Reward shaping dynamics We find a strong link between reward granularity and policy competence. Smaller models (8B) were especially responsive to shaped, process-level rewards, achieving performance competitive with 32B models while being up to 3.5\times more compute-efficient and 1.5\times more memory-efficient. Larger models (32B) performed well across reward settings and remained more robust under sparse signals, but exhibited higher variance. In contrast, 8B models depended more heavily on dense shaping. Curriculum learning alone provided no measurable benefit, whereas reward shaping consistently amplified learning dynamics, making the 8B models the most efficient setting for agentic RL. • Generalization Beyond Training Domain Our agents did not overfit to TravelPlanner : Planner-R1 models mostly maintained or exceeded baseline performance on out-of-domain tasks including Multi-IF , NaturalPlan , and \tau - Bench . This demonstrates that the efficiency gains from agentic RL come without sacrificing robustness, supporting transfer to diverse planning and tool-use settings. • RL Benchmark Formulation We recast TravelPlanner as a multi-step agentic RL benchmark by leveraging the official sandbox and its seven tools, and we designed verifiable reward functions aligned with the task’s success criteria. Policies were trained with verl ver 2024 , where our system-level optimizations reduced runtime and memory usage by 20%, enabling efficient large-scale experimentation. (see Appendix A for details) 2 Planner RL 2.1 Problem Formulation Figure 2: MDP Visualization. x_{i} represent the i th token, while a_{t} represents the action the agent took at time t . Notice that initial prompts and tool responses contain tokens, but they dont increase the time step t . We cast tool-augmented planning as a Markov Decision Process (MDP) \mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma) . Since our MDP is episodic, we set \gamma=1 . Each episode is initialized with two textual inputs: a system prompt y , which

Read the complete text