Seek in the Dark : Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space
2025
Read Seek in the Dark : Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13308.txt.
Introduction
Li and co-authors give language models a new way to think harder at answer time: instead of generating more words, search inside the model's own hidden states. Their framework, LatentSeek, treats reasoning as movement through latent space. When faced with a problem, it nudges the model's internal representation step by step, guided by reward signals the model generates itself, using policy gradient updates that leave the model's weights untouched.
On math and reasoning benchmarks, from grade-school word problems to competition questions, LatentSeek beat strong baselines including chain-of-thought prompting and fine-tuned rivals. It usually settled within a few adjustment rounds on average problems, and extra rounds kept helping on harder ones. The authors frame latent search as a cleaner way to spend test-time compute than longer text. For researchers chasing reasoning gains without retraining, the paper offers a lightweight dial worth turning.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13308
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Seek in the Dark : Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space Abstract marginparsep has been altered. topmargin has been altered. marginparpush has been altered. The page layout violates the ICML style. Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you. We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again. Seek in the Dark : Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space Hengli Li {}^{\,1,2\,*} , Chenxi Li {}^{\,2,3\,*} , Tong Wu 2 , Xuekai Zhu 2,4 , Yuxuan Wang 2 , Zhaoxin Yu 5 , Eric Hanchen Jiang 6 , Song-Chun Zhu 1,2,3 , Zixia Jia 2 , Ying Nian Wu {}^{\,6\,} ✉ and Zilong Zheng {}^{\,2\,} ✉ {}^{1\,} Institute for Artificial Intelligence, Peking University {}^{2\,} NLCo Lab, Beijing Institute for General Artificial Intelligence {}^{3\,} Department of Automation, Tsinghua University {}^{4\,} Shanghai Jiao Tong University {}^{5\,} Institute of Automation, Chinese Academy of Sciences {}^{6\,} University of California, Los Angeles [email protected], [email protected], [email protected], [email protected] † † footnotetext: * Equal Contributions. ✉ Corresponding author(s): Ying Nian Wu, Zilong Zheng. Reasoning ability, a core component of human intelligence, continues to pose a significant challenge for Large Language Models (LLMs) in the pursuit of AGI. Although model performance has improved under the training scaling law, significant challenges remain, particularly with respect to training algorithms—such as catastrophic forgetting—and the limited availability of novel training data. As an alternative, test-time scaling enhances reasoning performance by increasing test-time computation without parameter updating. Unlike prior methods in this paradigm focused on token space, we propose leveraging latent space for more effective reasoning and better adherence to the test-time scaling law. We introduce LatentSeek , a novel framework that enhances LLM reasoning through Test-Time Instance-level Adaptation (TTIA) within the model’s latent space . Specifically, LatentSeek leverages policy gradient to iteratively update latent representations, guided by self-generated reward signals. LatentSeek is evaluated on a range of reasoning benchmarks, including GSM8K, MATH-500, and AIME2024, across multiple LLM architectures. Results show that LatentSeek consistently outperforms strong baselines, such as Chain-of-Thought prompting and fine-tuning-based methods. Furthermore, our analysis demonstrates that LatentSeek is highly efficient, typically converging within a few iterations for problems of average complexity, while also benefiting from additional iterations, thereby highlighting the potential of test-time scaling in the latent space. These findings position LatentSeek as a lightweight, scalable, and effective solution for enhancing the reasoning capabilities of LLMs. Code https://github.com/bigai-nlco/LatentSeek Project https://bigai-nlco.github.io/LatentSeek/ Figure 1: Comparison of LatentSeek with RL-based fine-tuning and Prompt Engineering. RL-based fine-tuning methods generally require iterative updates to model parameters guided by reward signals. Prompt engineering approaches depend heavily on manually designed prompts. In contrast, LatentSeek performs optimization within the latent space. Of note, the output of LatentSeek may be incoherent and semantically ungrounded ( \lx@sectionsign 3.8 ). 1 Introduction Large Language Models (LLMs) have demonstrated exceptional performance across a wide array of tasks, particularly in complex reasoning and deductive analysis ( Brown et al., 2020 , Chowdhery et al., 2023 , OpenAI, 2023 , Zhao et al., 2025 ) . Despite these advancements, LLMs still exhibit difficulties in reasoning, particularly in tasks that demand structured thinking and meticulous step-by-step analysis ( Wei et al., 2022 , Kojima et al., 2022 ) . Common approaches to enhancing the reasoning capabilities of LLMs involve training model parameters on reasoning-centric datasets or providing reasoning-oriented feedback ( Ouyang et al., 2022 , Bai et al., 2022 , DeepSeek-AI, 2025 ) . However, training methodologies such as supervised fine-tuning, reinforcement learning, and test-time training Sun et al. (2020) , Hardt and Sun (2024) need to update model parameters, which incur substantial computational costs and present potential risks, including catastrophic forgetting of general competencies ( Luo et al., 2025 ) . Moreover, the widely used reinforcement learning approach may reduce the model’s exploration capacity Yue et al. (2025) and, in some cases, lead to the generation of overly verbose responses Aggarwal and Welleck (2025) , Wu et al. (2025) . In light of these challenges, we focus on an alternative paradigm, tta ( tta ) , that does not require parameter updating and operates on a per-instance basis during the testing phase . A prevalent approach in TTIA is prompt engineering, which suffers from limited expressiveness. To enhance TTIA performance in reasoning, recent advances Deng et al. (2022) , Hao et al. (2024) reveal that reasoning capabilities can be internalized within the latent space 1 1 1 In this work, we take the convention Kong et al. (2025) , Hao et al. (2024) that treats the transformers’ output space ahead of the final language model (LM) head as latent space ( Figure 1 ), and the vector in the space as latent representation ( Figure 1 ); refer to \lx@sectionsign 2.2 for notations. through fine-tuning. However, the training strategies adopted in these works tend to substantially modify the latent space and fail to fully leverage its semantic richness. As a result, their performance remains inferior to that of cot ( cot ). Nevertheless, these studies provide evidence supporting the adequacy of semantic information encoded within latent representations, i.e., the hidden states corresponding to language tokens. Motivated by these observations, we present the first attempt to perform seeking in the latent space by introducing LatentSeek , a framework that significantly enhances instance-level reasoning at test time. LatentSeek introduces updated instance-specific latent representations that steer the pre-trained model’s reasoning process without modifying its parameters. These latent representations act as a planning or control mechanism that guides the model toward better reasoning paths for each specific problem instance. We optimize latent representations at test time using the policy gradient method Williams (1992) to maximize reward ( \lx@sectionsign 2.3 ). Specifically, for each reasoning problem, we update the token-wise latent representations using guidance from the reward function, treating them as independent variables. In each iteration, the updated latent representations are decoded into tokens, which serve as inputs for computing the reward. Importantly, the reward function operates in a self-rewarding manner, relying solely on the model’s internal capabilities without incorporating any external information. The process continues until the reward exceeds a predefined threshold or the maximum number of iterations is reached. Our innovative latent space reasoning method is simple yet effective : Notably, LatentSeek yields average gains of 15.23 points over BoN on GSM8K, 4.72 points over Self-Reflection on MATH-500, and 6.67 points over cot on AIME2024 ( Table 1 ). Furthermore, when using LLaMA3.1-8B-Instruct as the backbone, LatentSeek surpasses prior arts including SimpleRL-Zoo ( Williams, 1992 ) ( +18.1 ) and Genius ( Xu et al., 2025a ) ( +12.7 ), on the GSM8K and MATH-500 datasets ( Table 2 ). Further experiments ( \lx@sectionsign 3.4 ) show that test-time performance improves with the number of update iterations, suggesting a complementary scaling axis: the number of optimization steps in latent space . To better characterize the latent space, we conduct idealized experiments using a perfect, ground-truth verifier that issues sharp rewards. Across all evaluation settings, this yields an average gain of 19.12 points over cot ( Table 3 ), highlighting the benefits of pure exploration in the latent