Inference-Time Budget Control for LLM Search Agents

Zhengru Fang et al · 2026

Read Inference-Time Budget Control for LLM Search Agents by Zhengru Fang et al (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05701.txt.

Introduction

Search agents spend two currencies at once: tool calls and tokens, both capped. Zhengru Fang and co-authors study how to budget those currencies during multi-hop question answering. Better answers need not only stronger models but decisions about which search action deserves the next unit of budget, and when the evidence is enough to stop.

Their controller scores each feasible action by value of information: marginal task value per unit budget, given the current state and what remains. A second stage decides when to commit a final answer. The formulation treats inference-time spending as an explicit optimization problem.

Catalog record

CREATORZhengru Fang et al
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05701

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Inference-Time Budget Control for LLM Search Agents Zhengru Fang et al Abstract LLM search agents increasingly rely on tools at inference time, but their trajectories are often constrained by hard limits on both tool calls and generated tokens. Under such dual budgets, better answers require not only stronger models, but also explicit control over which search action should receive the next budget unit and when the accumulated evidence is sufficient to commit a final answer. We study this problem in multi-hop question answering (QA) and formulate it as two-stage inference-time budget control. At search time, our controller assigns each feasible action a task-level Value-of-Information (VOI) score, defined as an operational estimate of marginal task value per unit budget under the current search state and remaining dual budget, and uses this score to choose among retrieval, decomposition, and answer commitment. After search, a selective evidence-grounded finalizer compares the trajectory answer with a refined candidate and rewrites only when the residual error appears to be a low-risk answer-form error. Across four multi-hop QA benchmarks, three LLM backbones, and four budget levels, the method yields positive aggregate gains over four audited baselines under the same hard dual-budget protocol. Ablations show that search-time budget control, especially budget-dependent penalty, provides the main performance gain, while answer-time control helps mainly when the retrieval path is already adequate. These results suggest that inference-time budget control for LLM search agents should govern both how budget is spent during search and how the final answer is committed. \linenumbers 1 Introduction Large language models (LLMs) are increasingly deployed as search agents that use external tools during inference. ReAct [ 69 ] , Toolformer [ 43 ] , BATS [ 28 ] , and Search-o1 [ 21 ] treat retrieval as part of the inference trajectory rather than a fixed preprocessing step. Broader agent systems add reflection, tree search, collaboration, and software tools [ 46 , 68 , 7 , 53 , 78 , 77 , 30 ] . In this paradigm, answer quality depends not only on model capability, but also on how the agent spends tool-call and token budgets while inference is still unfolding. This paradigm also changes test-time scaling from a token-only problem into an inference-time control problem. Prior work studies how additional inference compute improves outputs through repeated sampling, self-refinement, and adaptive allocation [ 56 , 32 , 47 , 12 , 25 ] . For search agents, however, compute is not token-only: extra budget can create more branches, intermediate states, and low-yield actions. Systems such as Search-R1 [ 17 ] , BATS [ 28 ] , and recent browsing agents [ 80 , 57 , 45 , 15 ] already operate in this tool-heavy regime. Deployment studies likewise show that unconstrained tool use leads to budget-inefficient failures, including redundant retrieval, wasted branching, and brittle context growth [ 6 , 72 , 19 , 31 ] . Under explicit budgets, more inference-time compute and better answers are no longer equivalent. This raises the central inference-time control problem of how to decide, under explicit tool-call and output-token budgets, what to do next, and how to commit a final answer once search stops. Existing budget-aware approaches either route across models to reduce cost [ 6 , 72 ] or inject budget tracking into the agent loop [ 28 , 74 ] , but none of them explicitly control both action allocation during search and answer resolution after search under the same hard dual-budget audit, which motivates this work. Three difficulties make this control problem nontrivial. Challenge 1: Budget must be allocated across heterogeneous actions. At each step a search agent can retrieve evidence, decompose the question, or answer directly. These actions differ in cost and return, and the right choice depends on the evidence state and remaining budget. Infrastructure analyses show that naive scaling leads to over-search and diminishing returns [ 19 , 3 ] , while token-level budgeting cannot resolve this action-level trade-off under coupled tool-call and output-token budgets [ 12 , 25 ] . Challenge 2: Final-answer errors persist after successful search. Even with the right evidence, the final answer can fail on exactness: yes/no polarity, binary choice, typed slots, or alias completion. Self-refinement [ 32 ] , stepwise verification [ 5 , 48 ] , and reasoning-aware selection [ 52 ] can help, but they target open-ended generation rather than budgeted search where every extra call has a cost. Challenge 3: Rewriting introduces intervention risk. Unconditional answer rewriting can erase bridge structure or reverse comparative semantics in multi-hop settings [ 51 , 50 ] . Reflexion-style feedback loops [ 46 ] mitigate some errors but do not explicitly model the risk of damaging a correct answer. Thus, an answer controller must intervene only when the expected gain outweighs risk. To address these challenges, we propose a training-free, two-stage inference-time controller built on a tree-search backbone inspired by BAVT [ 23 ] . At search time, the controller scores each feasible action by task-level VOI: an operational estimate of marginal task value per unit budget under the current trajectory state and remaining dual budget, rather than Shannon information gain or Bayesian posterior value. The score combines critic-derived value change, structural signals, cost normalization, budget penalty, and conservative guards to choose among retrieval, decomposition, and answer commitment. After search, an evidence-grounded finalizer rewrites only when expected exactness gain outweighs the risk of damaging an otherwise adequate trajectory answer. The method therefore controls two distinct decision points: how budget is spent during search, and how the final answer is committed after search. We evaluate VOI under a shared hard dual-budget audit across four multi-hop QA benchmarks, three LLM backbones, and a four-level budget ladder. The results support explicit two-stage control while also exposing its boundary conditions across datasets, budgets, and backbones. Our main contributions are as follows: ❶ Two-stage inference-time budget-control formulation. We formulate budget-aware inference in LLM search agents as a two-stage inference-time control problem: search-time action allocation under coupled tool-call and output-token budgets, followed by answer-time finalization under intervention risk. ❷ Task-level VOI control for search actions. We introduce a training-free scorer that ranks retrieval, decomposition, and answer commitment by estimated marginal task value per unit budget. The scorer combines critic-derived progress, structural signals, cost normalization, budget-dependent penalty, and conservative guards. ❸ Risk-controlled answer finalization. We introduce an evidence-grounded finalizer that rewrites only for low-risk answer-form errors, such as yes/no polarity, binary choices, typed-slot mismatches, or supported factoid completion, and abstains when bridge or comparative reasoning remains unresolved. ❹ Empirical analysis under hard dual-budget audit. We evaluate across four benchmarks, three backbones, and four budget levels under the same hard dual-budget protocol. The results show positive aggregate gains and many low- and mid-budget improvements, while ablations identify budget-dependent penalty as the dominant component and expose backbone- and dataset-dependent boundary conditions where the gains diminish. 2 Related Work Search agents and tool-augmented inference. Tool use shifted LLMs from static text generation to interactive decision-making. ReAct [ 69 ] interleaves reasoning and acting, Toolformer [ 43 ] studies self-supervised tool use, and later systems add reflection, tree search, collaboration, environment grounding, and software tools [ 46 , 68 , 7 , 29 , 53 , 78 , 77 ,

Read the complete text