RTTC : Reward-Guided Collaborative Test-Time Compute
J. Pablo MuñozIntel LabsSanta Clara, CA, USA [email protected]@gmail.com, Jinjie YuanIntel CorporationBeijing, China [email protected] · 2025
Read RTTC : Reward-Guided Collaborative Test-Time Compute by J. Pablo MuñozIntel LabsSanta Clara, CA, USA [email protected]@gmail.com, Jinjie YuanIntel CorporationBeijing, China [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10024.txt.
Introduction
Muñoz and Yuan published a research paper in 2025 on choosing, query by query, the cheapest way to make a language model think harder. Extra test-time effort can mean fetching documents or light fine-tuning, but the best choice varies and blanket use wastes compute. Their framework lets a pretrained reward model pick the strategy per question.
Clients pull relevant samples from a remote knowledge base and adapt only when the reward model says it pays. A caching scheme reuses past query states at both the retrieval and adaptation levels to cut repeat work. Across models and benchmarks the adaptive approach beat always-retrieving or always-tuning, which the authors read as proof that guidance, not brute effort, is the lever.
Catalog record
| CREATOR | J. Pablo MuñozIntel LabsSanta Clara, CA, USA [email protected]@gmail.com, Jinjie YuanIntel CorporationBeijing, China [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-10024
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RTTC : Reward-Guided Collaborative Test-Time Compute J. Pablo MuñozIntel LabsSanta Clara, CA, USA [email protected]@gmail.com, Jinjie YuanIntel CorporationBeijing, China [email protected] Abstract Test-Time Compute (TTC) has emerged as a powerful paradigm for enhancing the performance of Large Language Models (LLMs) at inference, leveraging strategies such as Test-Time Training (TTT) and Retrieval-Augmented Generation (RAG). However, the optimal adaptation strategy varies across queries, and indiscriminate application of TTC strategy incurs substantial computational overhead. In this work, we introduce R eward-Guided T est- T ime C ompute ( RTTC ), a novel framework that adaptively selects the most effective TTC strategy for each query via a pretrained reward model, maximizing downstream accuracy across diverse domains and tasks. RTTC operates in a distributed server-client architecture, retrieving relevant samples from a remote knowledge base and applying RAG or lightweight fine-tuning on client devices only when necessary. To further mitigate redundant computation, we propose Query-State Caching, which enables the efficient reuse of historical query states at both retrieval and adaptation levels. Extensive experiments across multiple LLMs and benchmarks demonstrate that RTTC consistently achieves superior accuracy compared to vanilla RAG or TTT, validating the necessity of adaptive, reward-guided TTC selection and the potential of RTTC for scalable, high-performance language model adaptation. 1 Introduction Figure 1: Performance overview of RTTC on downstream tasks across various LLMs. “Accuracy Improvement” indicates the relative average score on five downstream tasks evaluated in § 4 . TTC Strategy Latency Memory Accuracy No Adaptation (Direct Inference) + + + Retrieval-Augmented Generation (RAG) ++ ++ +++ Test-Time Training (TTT) +++ +++ +++ Table 1: Comparison of computational cost and accuracy boosting benefits for different Test-Time Compute (TTC) strategies. More plus signs indicate higher cost or increased accuracy. While RAG and TTT both provide significant accuracy improvements, each has its strengths, depending on the query and scenario. Large language models have achieved remarkable performance across a wide range of tasks. However, their robustness and adaptability to new domains or distribution shifts at inference time remain open challenges. Traditionally, two major paradigms have been explored to enhance LLM performance at test time: Retrieval-Augmented Generation (RAG) Lewis et al. 2020 ; Gao et al. 2023b , which augments model input with retrieved knowledge, and Test-Time Training (TTT) Sun et al. 2020 ; Hardt and Sun 2024 ; Hübotter et al. 2024 ; Snell et al. 2025 , which adapts model parameters using relevant samples. Both approaches have demonstrated significant effectiveness. Figure 2: Workflow of R eward-guided T est- T ime C ompute ( RTTC ). For each query, a pretrained reward model evaluates candidate responses and selects the optimal adaptation strategy (No Adaptation, RAG, or TTT). \tau_{r} is a predefined threshold. However, the practical deployment of RAG and TTT raises two concerns: accuracy and efficiency. Accuracy: The effectiveness of RAG or TTT also varies across queries. Sometimes RAG outperforms TTT, and vice versa. For some inputs, the model’s direct response is already sufficiently accurate, while for others, the inference stage may benefit from retrieval augmentation or adaptive fine-tuning. Efficiency: Both RAG and TTT introduce significant computational overheads—RAG increases inference latency and memory usage by expanding the input context, while TTT requires additional fine-tuning steps and memory for model updates. Moreover, naively applying RAG or TTT to every query can lead to unnecessary computation and inefficient resource utilization. A summary of the computational cost and accuracy associated with each strategy is provided in Table 1 . Therefore, an adaptive approach that can dynamically select the optimal strategy at test-time for each query is crucial for maximizing performance while minimizing overhead . Aiming to tackle the above challenges, we propose R eward-guided T est- T ime C omputing ( RTTC ). This framework dynamically selects among three strategies for each query: No Adaptation (i.e., returning the model’s response without adaptation), Retrieval-Augmented Generation (RAG), and Test-Time Training (TTT). At the core of RTTC is a pretrained reward model that evaluates candidate responses and guides the system to choose the most effective adaptation strategy in a query-adaptive manner (see Figure 2 ). This reward-guided collaboration enables RTTC to adaptively exploit the most suitable strategy for each query, achieving robust downstream performance improvements across diverse domains and tasks. Unlike prior work that statically applies either RAG or TTT, our approach introduces a principled decision-making mechanism that maximizes performance. Additionally, RTTC also introduces the Query-State Caching ( QSC ) mechanism to further optimize the test-time efficiency. QSC leverages historical query embeddings and their associated retrieved samples or fine-tuned model state (e.g., LoRA Hu et al. 2022 ) to potentially bypass the need for repeated retrieval and fine-tuning, thus reducing computational overhead and latency. Overall, our main contributions are: 1. We introduce RTTC , a reward-guided collaborative test-time compute framework. We design an effective decision process that leverages a pre-trained reward model to adaptively choose the optimal inference strategy, enabling robust LLM adaptation. 2. We further propose a Query-State Caching (QSC) mechanism that reuses historical query information, reducing redundant computation and latency during inference. 3. Extensive experiments demonstrate that RTTC consistently outperforms baselines, achieving higher accuracy across multiple LLMs and downstream tasks. Figure 3: Overview of the retrieval and test-time compute stages. When the reward model determines that the LLM response without adaptation does not meet expectations, relevant samples are retrieved from a remote multi-domain knowledge base. By leveraging advanced test-time compute strategies (RAG or TTT), RTTC improves model performance on client devices. In summary, our work is the first to unify direct inference, RAG, and TTT within a reward-guided, query-adaptive framework. The remainder of the paper is organized as follows. We discuss related work in § 2 . Then, § 3 describes the RTTC system, while § 4 presents results on various downstream tasks. Our final thoughts are in § 5 . 2 Related Work Test-Time Compute. Test-time compute techniques have been proposed as an alternative to scaling model parameters for improving model performance Snell et al. 2025 . Notable strategies include Chain-of-Thought (CoT) prompting Wei et al. 2022 and few-shot learning Brown et al. 2020 . CoT prompting guides the model through intermediate steps to break down complex tasks, enhancing its ability to handle intricate queries. Few-shot learning helps the model adapt to new tasks with just a few examples. Other test-time compute methods include verifying the model’s results, for example, through code execution Brown et al. 2025 . Retrieval-Augmented Generation (RAG). RAG has emerged as a prominent paradigm for enhancing model performance by incorporating external knowledge at inference time Lewis et al. 2020 ; Gao et al. 2023b . By retrieving relevant documents and augmenting the model’s input, RAG enables LLMs to access up-to-date or domain-specific information beyond their pretraining corpus. This approach has demonstrated strong results across open-domain question answering, fact verification, and knowledge-intensive tasks. RAG introduces additional computational overhead due to retrieval and longer input sequences, and its effectiveness can vary depending on the quality of retrieved content and the task. Test-Time Training (TTT). TTT Hardt and Sun 2024 ; Hübotter et al. 2024 ; Akyürek et al. 2024 has proven effective for adapting models to distribution shifts by fine-tuning on retrieved samples during inference. Recent work, such as SIFT Hübotter et al. 2024 , has improved retrieval strategies for TTT,