Recursive Agent Optimization

Apurva Gandhi, Satyaki Chakraborty, Xiangjun Wang, Aviral Kumar, Graham Neubig · 2026

Read Recursive Agent Optimization by Apurva Gandhi, Satyaki Chakraborty, Xiangjun Wang, Aviral Kumar, Graham Neubig (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06639.txt.

Introduction

Apurva Gandhi, Satyaki Chakraborty, Xiangjun Wang, Aviral Kumar, and Graham Neubig introduce Recursive Agent Optimization, a reinforcement learning method for agents that can spawn copies of themselves. A recursive agent delegates sub-tasks to fresh instances of itself, giving a divide-and-conquer way to scale past one context window and toward harder problems at inference time.

RAO trains models to use that ability well: when to delegate, how to split work, and how to communicate results back. Agents trained this way learn more efficiently, handle tasks longer than a single context, and generalize to problems much harder than their training tasks. They can also finish faster in wall-clock time than single-agent systems by working branches in parallel.

Catalog record

CREATORApurva Gandhi, Satyaki Chakraborty, Xiangjun Wang, Aviral Kumar, Graham Neubig
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06639

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Recursive Agent Optimization Apurva Gandhi, Satyaki Chakraborty, Xiangjun Wang, Aviral Kumar, Graham Neubig Abstract We introduce Recursive Agent Optimization (RAO) , a reinforcement learning approach for training recursive agents : agents that can spawn and delegate sub-tasks to new instantiations of themselves recursively. Recursive agents implement an inference-time scaling algorithm that naturally allows agents to scale to longer contexts and generalize to more difficult problems via divide-and-conquer. RAO provides a method to train models to best take advantage of such recursive inference, teaching agents when and how to delegate and communicate. We find that recursive agents trained in this way enjoy better training efficiency, can scale to tasks that go beyond the model’s context window, generalize to tasks much harder than the ones the agent was trained on, and can enjoy reduced wall-clock time compared to single-agent systems. 1 1 1 Webpage and code: https://apga.github.io/RAO 1 Introduction Large language model (LLM) agents are increasingly deployed on real-world tasks such as software engineering, research assistance, and computer use. As these systems improve, the tasks users expect them to solve are also becoming harder: they involve longer horizons, larger effective working memory, and substantial exploration and backtracking. Many such tasks are naturally amenable to divide-and-conquer. For instance, a large software change can be decomposed into code investigation and editing subproblems; a research task can be split into retrieval, synthesis, and verification stages; a long document or log corpus can be partitioned into manageable pieces and processed in parallel. This has led to interest in recursive agent systems ( Anthropic 2025 ; OpenAI 2026 ; Sun et al. 2025 ; Zhang et al. 2025 ) , in which an agent can spawn sub-agents to solve subtasks with fresh contexts. Recursive execution offers several advantages. First , it expands effective working memory, since each sub-agent receives a fresh context window rather than inheriting the full history of its parent. Second , it enables divide-and-conquer by breaking a large task into smaller, more tractable subproblems. Third , when subproblems can be solved independently, recursion can exploit concurrency and reduce wall-clock latency for accomplishing a task. Together, these properties make recursion a compelling inference-time scaling primitive for agents. However, most existing recursive and multi-agent systems treat recursion purely as an inference-time scaffold wrapped around a pretrained model ( Prasad et al. 2024 ; Geng & Neubig 2026 ; Wu et al. 2023 ) . The model itself is typically not trained to decide when delegation is useful, how to formulate effective subtasks, how to communicate information across levels of the execution tree, or how to combine sub-agent outputs into a final solution. If recursive execution is going to be a core test-time primitive, then the policy should be trained to use it well. As such, prior work shows training models with inference-time scaffolds in the loop does make them more amenable to deployment with inference-time scaffolds ( Wu et al. 2026 ; LM-Provers et al. 2026 ; Kimi Team et al. 2026 ) . This raises a central question: how should we train a model to exploit recursive inference effectively? In this work, we introduce Recursive Agent Optimization (RAO) , a reinforcement learning approach for end-to-end training of recursive agents. RAO trains a single LLM policy that is instantiated at every node of a recursively-generated execution tree. As a result, the same model must learn both how to solve the task assigned to it and how to generate useful delegated subtasks for spawned copies of itself. Importantly, RAO is not restricted to a fixed hierarchy or hand-designed orchestration scheme: it supports dynamically generated recursive execution trees and optimizes all decisions in the tree jointly. Training recursive agents may also improve learning efficiency, not just inference-time behavior. Recursive execution naturally generates related tasks at multiple levels of difficulty, yielding structured intermediate supervision and an implicit curriculum over simpler subproblems. Thus, the recursive structure used at inference time can also provide a useful training signal for generally improving model capability. This motivates a second question that we tackle: how can we leverage the recursive structure of inference to better train agents? We evaluate RAO on three benchmarks, including deep research, long document processing and TextCraft-Synth , a controlled synthetic environment we introduce, inspired by Minecraft-style crafting tasks, in which we can vary task horizon, difficulty, and recursive structure. We find that training for recursive execution yields several benefits. Recursive agents trained with RAO solve tasks that exceed the base model’s context window, generalize to substantially harder tasks than those seen during training by leveraging recursive delegation (up to 10 levels deep), and when tasks can be decomposed into parallel subproblems, can reduce wall-clock execution time relative to non-recursive baselines (up to 2.5 \times ). In addition, RAO improves training efficiency over single-agent training by exploiting recursive decomposition during learning. Figure 1: Example of recursive agent inference on a deep research or travel-planning task where agents can decide to spawn and delegate sub-problems to sub-agents. This produces a dynamically structured and recursive execution tree where each node corresponds to one agent instance attempting to solve an assigned task. RAO optimizes a single policy to act across all levels tree, teaching it to best take advantage of recursive inference. 2 Recursive Agents: Inference and Training We study recursive agents : agents that can delegate sub-tasks to new instances of themselves during execution. A rollout no longer resembles a single flat trajectory, but a dynamically generated tree of trajectories. While many prior works and deployed systems have explored recursive inference (Section 6 ), this work focuses on how to best train a model to leverage recursion: it must learn how to solve assigned tasks directly, when delegation is beneficial, how to formulate sub-tasks for child agents, and how to combine their outputs into a final solution. Unlike approaches based on fixed hierarchies or hand-specified orchestration, our setting allows the policy to generate execution trees dynamically, with arbitrary branching patterns up to a depth limit, and RAO trains all agent instances in the tree jointly. 2.1 Recursive Agent Inference Formulation. A task X is a problem specification given to an agent. When the policy \pi_{\theta} is run on X , it produces a trajectory \tau_{X} , consisting of the sequence of actions and observations used to solve the task. During this rollout, the agent may decide to spawn zero or more child agents on delegated sub-tasks X_{1},\dots,X_{n} , where both the number and the content of these sub-tasks are chosen by the policy itself. This process induces a rooted execution tree \mathcal{T} : each node corresponds to one agent instance attempting one assigned task, the root corresponds to the original task, and the children of a node correspond to the delegated sub-tasks generated by that agent. Fig. 1 shows an example of a recursive agent trajectory on a travel-planning task. In particular, note that sub-agents can also recursively spawn their own child agents. Implementation. We instantiate recursive agents as an extension of an agent that interleaves natural-language reasoning with code execution in a Python REPL (read-eval-print-loop) similar to Wang et al. 2024 . This interface gives the model access to standard programming constructs such as variables, control flow, string manipulation, and asynchronous

Read the complete text