Dynamic Speculative Agent Planning
Yilin Guan 1 Qingfeng Lan 2 Fei Sun Dujian Ding 3 Devang Acharya 4 Chi Wang 5 William Yang Wang 6 Wenyue Hua 6 1 Johns Hopkins University 2 University of Alberta 3 University of British Columbia 4 Avey Research Center 5 Google DeepMind 6 University of California, Santa Barbara {yguan29}@jh.edu, weny · 2025
Read Dynamic Speculative Agent Planning by Yilin Guan 1 Qingfeng Lan 2 Fei Sun Dujian Ding 3 Devang Acharya 4 Chi Wang 5 William Yang Wang 6 Wenyue Hua 6 1 Johns Hopkins University 2 University of Alberta 3 University of British Columbia 4 Avey Research Center 5 Google DeepMind 6 University of California, Santa Barbara {yguan29}@jh.edu, weny (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01920.txt.
Introduction
Yilin Guan leads seven co-authors with Dynamic Speculative Planning, a method that speeds up language model agents without losing quality. Agents built on large models handle hard tasks but run slowly and cost much to use. Past speedups either lost fidelity, needed heavy offline training, or cost too much to run. Users also had little say over the speed versus cost balance.
Their framework learns online while it works and balances total delay against dollar cost in one objective. A single knob steers the system toward faster, cheaper, or middle settings. On two standard agent benchmarks, it matches the fastest lossless rival on efficiency. Total cost falls by about a third, and needless cost falls by up to three fifths.
Catalog record
| CREATOR | Yilin Guan 1 Qingfeng Lan 2 Fei Sun Dujian Ding 3 Devang Acharya 4 Chi Wang 5 William Yang Wang 6 Wenyue Hua 6 1 Johns Hopkins University 2 University of Alberta 3 University of British Columbia 4 Avey Research Center 5 Google DeepMind 6 University of California, Santa Barbara {yguan29}@jh.edu, weny |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-01920
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dynamic Speculative Agent Planning Yilin Guan 1 Qingfeng Lan 2 Fei Sun Dujian Ding 3 Devang Acharya 4 Chi Wang 5 William Yang Wang 6 Wenyue Hua 6 1 Johns Hopkins University 2 University of Alberta 3 University of British Columbia 4 Avey Research Center 5 Google DeepMind 6 University of California, Santa Barbara {yguan29}@jh.edu, [email protected] Abstract Despite their remarkable success in complex tasks propelling widespread adoption, large language-model-based agents still face critical deployment challenges due to prohibitive latency and inference costs. While recent work has explored various methods to accelerate inference, existing approaches suffer from significant limitations: they either fail to preserve performance fidelity, require extensive offline training of router modules, or incur excessive operational costs. Moreover, they provide minimal user control over the tradeoff between acceleration and other performance metrics. To address these gaps, we introduce Dynamic Speculative Planning (DSP), an asynchronous online reinforcement learning framework that provides lossless acceleration with substantially reduced costs without requiring additional pre-deployment preparation. DSP explicitly optimizes a joint objective balancing end-to-end latency against dollar cost, allowing practitioners to adjust a single parameter that steers the system toward faster responses, cheaper operation, or any point along this continuum. Experiments on two standard agent benchmarks demonstrate that DSP achieves comparable efficiency to the fastest lossless acceleration method while reducing total cost by 30% and unnecessary cost up to 60%. Our code and data are available through https://github.com/guanyilin428/Dynamic-Speculative-Planning . 1 Introduction LLM - based agents have rapidly moved from controlled demonstrations to real - world deployments, supporting complex multi - step tasks in domains such as autonomous software engineering ( Wu et al., 2023 ) , open - world tool use ( Shen et al., 2024 ) , and everyday personal assistance ( Ge et al., 2024 ) . Although LLM - based agents have achieved widespread adoption, practitioners consistently report substantial end - to - end latency ( Hua et al., 2024 ; Zhang et al., 2024b ; Zhang et al., 2023 ; Saha et al., 2024 ) . This latency not only degrades user experience but also limits the feasibility of deploying LLM - based agents in time - sensitive scenarios such as real - time decision support ( Cai et al., 2025 ) or interactive tutoring ( Modran et al., 2024 ; Piro et al., 2024 ) . Multiple lines of research attempt to address the latency bottlenecks. These include approaches such as context length reduction ( Zhang et al., 2024b ) , step parallelization ( Hua et al., 2024 ; Li et al., 2025 ; Pan et al., 2025 ) , and dual-process thinking that combines fast and deliberate reasoning modes ( Saha et al., 2024 ; Lin et al., 2023 ; Christakopoulou et al., 2024 ) . While these approaches offer various improvements in efficiency, there are several common critical limitations: (1) No guarantee on lossless performance except ISP: Saha et al., 2024 requires training a router between two agents and Zhang et al., 2024b requires careful prompt engineering to enable runtime tool-selection, both relying on the quality of the additional component. Pan et al., 2025 requires the model to learn to generate parallelizable steps. (2) Pre-deployment preparation : Saha et al., 2024 ; Pan et al., 2025 require training specialized modules in the system before deployment, adding complexity and overhead to implementation. (3) Lack of user-controllability ( Kiefer et al., 2017 ) : Most existing approaches neglect user-controlled flexibility in navigating efficiency improvements. This inflexibility becomes particularly problematic in the rapidly evolving LLM ecosystem, where pricing structures, model inference speeds, and model performance change frequently, and organizations have diverse priorities regarding latency versus operational costs including router training, prompt engineering, or extra cost brought by parallelizing computations. In this paper, we propose Dynamic Speculative Planning (DSP), a lossless, user-controllable agent planning acceleration framework that requires no pre-deployment preparation. In addition to ensuring lossless agentic performance via speculative strategy ( Hua et al., 2024 ) , DSP introduces three core improvements for cost reduction and user-controllability: First, we investigate the Pareto frontier of the latency-cost trade-off in speculative agent planning, determining the optimal cost for any given latency reduction target. This trade-off is governed by the speculation step , which refers to the number of future steps that an approximation agent attempts to predict ahead of verification. It determines how aggressively the system parallelizes computation to reduce latency. Our analysis reveals that the commonly adopted fixed-speculation-step approach ( Hua et al., 2024 ; Leviathan et al., 2023 ) is fundamentally limited: for complex tasks, aggressive speculation produces excessive and redundant agent calls that substantially increase costs, while for simpler tasks, conservative speculation fails to deliver sufficient acceleration. Since optimal speculation step is highly context-dependent and unknown a priori, fixed steps significantly constrain practical utility. Second, we develop a lightweight adaptive speculation step predictor that dynamically determines when to suspend speculation, effectively eliminating unnecessary cost while preserving acceleration benefits. This predictor employs online reinforcement learning that requires no external datasets, preprocessing steps, or dedicated pre-deployment phases. Instead, the system learns to optimize speculation step organically as it processes tasks, becoming increasingly efficient over time with zero additional infrastructure costs while ensuring immediate deployment benefits. Third, based on this dynamic prediction module, we introduce two mechanisms that modulate the trade-off between latency and cost : (1) biased step prediction and (2) unbiased step with biased offset, allowing practitioners to precisely calibrate system behavior anywhere along the spectrum from low-cost/low-speed to high-cost/high-speed operation. By providing fine-grained user control over the latency-cost balance, our framework accommodates diverse organizational priorities and adapts to the fluctuating pricing structures and inference speeds of the rapidly evolving LLM ecosystem. Predictor Approximation Target Trainer Action execution Canceled operations Predictor k=2 k=3 Approximation A \downarrow B \downarrow Target s_{1} s_{2} s_{3} A ✓ \downarrow C’ \downarrow D’ \downarrow B ✓ \downarrow Time C ✗ \downarrow New data for Training: ( s=s_{1},k=3 ) ( s=s_{2},k=2 ) ( s=s_{3},k=1 ) Trainer k=2 Cancel ongoing threads D \downarrow E \downarrow D ✓ \downarrow E ✓ \downarrow s_{5} Total task time s_{4} episode 1 episode 2 Figure 1 : Dynamic Speculative Planning with Online Reinforcement Learning. The diagram illustrates how DSP adaptively adjusts speculation steps across planning episodes. (1) Predictor dynamically infers step values k from current state s_{i} , while the Approximation ({\mathbf{A}}) and Target ({\mathbf{T}}) agents execute parallelly. (2) When mismatches occur (denoted by ✗ ), ongoing threads are canceled and execution resumes from the last verified step. (3) An asynchronous Trainer collects state-step ( s,k ) pairs from completed episodes, continuously updating the predictor without blocking execution. The structure of our system is presented in Figure 1 . Our empirical evaluation across two standard agent benchmarks demonstrate that DSP achieves the efficiency of the fastest lossless acceleration methods, while reducing total cost by up to 30%, cutting unnecessary cost by 60%, and preserving comparable acceleration, demonstrating the effectiveness of the framework. 2 Related Work Latency in LLM-based Agent LLM-based agents have gained widespread adoption in real-world applications ( Ge et al., 2024 ; Wu et al., 2023 ) due to their strong reasoning and planning capabilities. However, significant latency issues have been documented across multiple domains: Jaech et al., 2024 reports