Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scaling

Xinglin Wang, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Yiwei Li, Yueqi Zhang, Chuyi Tan, Ji Zhang, Boyuan Pan, Yao Hu, Kan Li · 2026

Read Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scaling by Xinglin Wang, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Yiwei Li, Yueqi Zhang, Chuyi Tan, Ji Zhang, Boyuan Pan, Yao Hu, Kan Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21684.txt.

Introduction

Xinglin Wang, Jiayi Shi, and eight co-authors recycle search effort to make test-time reasoning cheaper. Giving a model more compute at answer time broadens its search for a solution. Standard search treats each trial as disposable, throwing away the partial insights inside it.

The authors save and reuse that experience across trials. Lessons from dead ends can steer the next attempt away from the same walls. Reuse turns a pile of failed tries into a growing asset. More solved problems per unit of compute is the goal, which matters as reasoning features scale to everyday use.

Catalog record

CREATORXinglin Wang, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Yiwei Li, Yueqi Zhang, Chuyi Tan, Ji Zhang, Boyuan Pan, Yao Hu, Kan Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21684

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scaling Xinglin Wang, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Yiwei Li, Yueqi Zhang, Chuyi Tan, Ji Zhang, Boyuan Pan, Yao Hu, Kan Li Abstract Test-Time Scaling enhances the reasoning capabilities of Large Language Models by allocating additional inference compute to broaden the exploration of the solution space. However, existing search strategies typically treat rollouts as disposable samples, where valuable intermediate insights are effectively discarded after each trial. This systemic memorylessness leads to massive computational redundancy, as models repeatedly re-derive discovered conclusions and revisit known dead ends across extensive attempts. To bridge this gap, we propose Recycling Search Experience (RSE) , a self-guided, training-free strategy that turns test-time search from a series of isolated trials into a cumulative process. By actively distilling raw trajectories into a shared experience bank, RSE enables positive recycling of intermediate conclusions to shortcut redundant derivations and negative recycling of failure patterns to prune encountered dead ends. Theoretically, we provide an analysis that formalizes the efficiency gains of RSE, validating its advantage over independent sampling in solving complex reasoning tasks. Empirically, extensive experiments on HMMT24, HMMT25, IMO-Bench, and HLE show that RSE consistently outperforms strong baselines with comparable computational cost, achieving state-of-the-art scaling efficiency. 1 1 1 Our code and data have been released on https://github.com/WangXinglin/RSE . 1 Introduction “Wisdom comes from experience. Experience is often a result of lack of wisdom.” — Terry Pratchett Scaling parameters and data during pre-training has long been the primary driver of LLM performance, yet recent breakthroughs highlight a paradigm shift towards Test-Time Scaling (TTS) ( 4 ) . By leveraging extended computation at test time, models can effectively tackle complex reasoning tasks that were previously out of reach ( 26 ; 7 ) . TTS approaches can be broadly divided into two categories: Internal TTS, which trains the LLMs to “think” slowly with long Chain-of-Thought (CoT) ( 34 ; 14 ) , and External TTS, which improves performance by allocating additional inference-time compute with a fixed LLM ( 36 ; 45 ) . Focusing on the latter, test-time search has emerged as a prominent paradigm, where the model samples extensive rollouts to broaden the exploration of the solution space ( 50 ; 21 ) . Figure 1 : From memoryless rollouts to experience-guided search. (Left) Existing test-time scaling paradigms (parallel, sequential, and hybrid) largely treat rollouts as disposable: intermediate conclusions are repeatedly re-derived and dead ends are revisited across rollouts. (Right) Recycling Search Experience (RSE) runs rollouts in batches, distills reusable trajectory information into a shared Experience Bank, and conditions subsequent exploration on it. Positive recycling shares intermediate conclusions (e.g., \mathbf{C_{1}} – \mathbf{C_{3}} ) to shortcut redundant derivations, while negative recycling records failure patterns (e.g., \mathbf{D_{1}} ) to prune known dead ends. While yielding promising performance gains, existing test-time search strategies share a common bottleneck in information utilization: rollouts are often treated as disposable attempts rather than experience to be distilled and reused (See Figure 1 ). Specifically, parallel scaling expands breadth via extensive independent rollouts, but typically offers limited cross-branch sharing of intermediate insights ( 41 ; 18 ; 16 ; 39 ) ; sequential scaling iteratively improves a single draft, yet the information it accumulates is confined to unstructured in-context history: finite context windows force truncation or compression of earlier revisions, and the search remains a path-dependent local refinement around the current draft rather than a reusable pool of experience ( 23 ; 35 ; 5 ) ; current hybrid scaling uses external process reward models ( 36 ; 3 ; 45 ; 21 ; 40 ) or look-ahead evaluation ( 48 ; 37 ; 32 ) to allocate rollout budgets across candidate prefixes: expanding selected prefixes while pruning low-value branches. However, this prefix reuse is mostly within-branch: parallel branches rarely share trajectory insights (e.g., verified facts or failure causes), leaving experience from pruned paths unused and surviving branches to explore in isolation. Ultimately, the search process remains largely memoryless at the systemic level, causing the model to repeatedly re-derive the same intermediate steps and revisit similar dead ends across extensive rollouts, leading to substantial redundant computation. To address this inefficiency, we propose Recycling Search Experience (RSE) , a self-guided, training-free strategy that turns rollouts from disposable samples into reusable experience to actively guide the search process. Instead of viewing rollouts as isolated trials, we treat search as a cumulative process, where valuable insights from prior trajectories are fed back to guide later exploration. Leveraging the model’s self-assessment capability ( 43 ; 9 ) , RSE distills valuable insights from rollouts into a shared experience bank without relying on external supervision, and explicitly conditions subsequent exploration on this bank. This mechanism enables two forms of experience reuse: positive recycling , where intermediate conclusions are shared to shortcut redundant derivations, and negative recycling , where encountered dead ends serve as constraints to prune the search space. By iterating this distill-and-guide loop, RSE turns test-time scaling from isolated trials into a cumulative, experience-guided search that concentrates compute on promising, unexplored regions of the solution space. Intuitively, solving complex reasoning problems via independent sampling requires a single rollout to sequentially derive all necessary intermediate conclusions—a probability that decays exponentially with problem complexity. In contrast, RSE effectively “checkpoints” valid intermediate conclusions, allowing the model to piece together the solution from fragmented partial successes rather than starting from scratch. We provide a theoretical analysis that formalizes and validates this intuition (see Appendix A ). To validate the effectiveness of RSE, we evaluate it on challenging mathematical reasoning benchmarks, including HMMT24, HMMT25 ( 2 ) , IMO-Bench ( 22 ) , and HLE ( 33 ) , across a broad range of rollout budgets and policy models. The empirical results show that RSE consistently outperforms strong baseline strategies under comparable computational budgets, highlighting its ability to enhance the overall efficiency of TTS. To summarize, our main contributions are: • We propose Recycling Search Experience (RSE), a training-free inference strategy that turns test-time search into a cumulative, experience-guided process by recycling rollout-level experience. • We provide a theoretical analysis that formalizes the efficiency gains of RSE, validating the advantages of experience recycling over independent sampling in solving complex reasoning tasks. • We empirically validate RSE on challenging mathematical reasoning benchmarks (HMMT24, HMMT25, IMO-Bench, HLE), demonstrating consistent gains over strong baselines across diverse models and rollout budgets. 2 Related Work Test-Time Scaling. The paradigm of scaling test-time compute has emerged as a critical avenue for enhancing reasoning capabilities ( 26 ; 7 ) . Early approaches explored two primary dimensions: scaling depth via sequential refinement ( 42 ; 23 ; 35 ) , which iteratively improves a single chain of thought; and scaling width via ensemble sampling ( 41 ) , which leverages the diversity of independent rollouts to marginalize out errors. To add structure beyond depth/width scaling, prior work explores tree-based search such as Tree of Thoughts ( 48 ) and MCTS ( 19 ; 37 ) , which allocate compute via explicit lookahead and rely on external supervision (e.g., labels or outcome reward models) to estimate the value of intermediate states. More recently, Process Reward Models (PRMs) provide step-level feedback for pruning and

Read the complete text