TEMPO: Scaling Test-time Training for Large Reasoning Models

Qingyang Zhang, Xinke Kong, Haitao Wu, Qinghua Hu, Minghao Wu, Baosong Yang, Yu Cheng, Yun Luo, Ganqu Cui, Changqing Zhang, Tianjin University Tongyi Lab, Alibaba Group The Chinese University of Hong Kong Shanghai AI Lab · 2026

Read TEMPO: Scaling Test-time Training for Large Reasoning Models by Qingyang Zhang, Xinke Kong, Haitao Wu, Qinghua Hu, Minghao Wu, Baosong Yang, Yu Cheng, Yun Luo, Ganqu Cui, Changqing Zhang, Tianjin University Tongyi Lab, Alibaba Group The Chinese University of Hong Kong Shanghai AI Lab (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19295.txt.

Introduction

Zhang and colleagues tackle a stall in test-time training, where a reasoning model keeps learning from unlabeled test questions during inference. Earlier methods plateau and lose diversity because their self-made reward signal drifts without outside calibration. Their framework, TEMPO, alternates policy refinement on unlabeled questions with critic recalibration on labeled data. Framed through expectation-maximization, the method restores the missing recalibration step and tightens the training bound.

Catalog record

CREATORQingyang Zhang, Xinke Kong, Haitao Wu, Qinghua Hu, Minghao Wu, Baosong Yang, Yu Cheng, Yun Luo, Ganqu Cui, Changqing Zhang, Tianjin University Tongyi Lab, Alibaba Group The Chinese University of Hong Kong Shanghai AI Lab
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38915
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19295

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TEMPO: Scaling Test-time Training for Large Reasoning Models Qingyang Zhang, Xinke Kong, Haitao Wu, Qinghua Hu, Minghao Wu, Baosong Yang, Yu Cheng, Yun Luo, Ganqu Cui, Changqing Zhang, Tianjin University Tongyi Lab, Alibaba Group The Chinese University of Hong Kong Shanghai AI Lab Abstract Test-time training (TTT) adapts model parameters on unlabeled test instances during inference time, which continuously extends capabilities beyond the reach of offline training. Despite initial gains, existing TTT methods for LRMs plateau quickly and do not benefit from additional test-time compute. Without external calibration, the self-generated reward signal increasingly drifts as the policy model evolves, leading to both performance plateaus and diversity collapse. We propose TEMPO , a TTT framework that interleaves policy refinement on unlabeled questions with periodic critic recalibration on a labeled dataset. By formalizing this alternating procedure through the Expectation-Maximization (EM) algorithm, we reveal that prior methods can be interpreted as incomplete variants that omit the crucial recalibration step. Reintroducing this step tightens the evidence lower bound (ELBO) and enables sustained improvement. Across diverse model families (Qwen3 and OLMO3) and reasoning tasks, TEMPO improves OLMO3-7B on AIME 2024 from 33.0% to 51.1% and Qwen3-14B from 42.3% to 65.8%, while maintaining high diversity. Code is available at this url . Project Page | GitHub | HuggingFace Figure 1: Scalability of TEMPO on the AIME benchmark. TEMPO alternates between an E-Step (critic recalibration on labeled data) and an M-Step (policy refinement on unlabeled test questions), guided by a critic that provides quality-aware scores. Representative self-rewarding TTT baselines such as TTRL (grey curves) plateau and collapse after initial gains. In contrast, TEMPO (blue curve) sustains a consistent upward trajectory over 350 steps by periodically grounding the critic in external supervision, demonstrating that additional test-time compute translates into scalable performance gains on complex open problems. 1 Introduction Large reasoning models (LRMs) have demonstrated remarkable capabilities on complex reasoning tasks, including logic puzzle-solving and Olympiad-level mathematics and physics 4 ; 3 ; 11 . These models achieve their performance by allocating extensive computation at test time through extended reasoning chains. However, their parameters remain static after training, which prevents them from incorporating knowledge acquired during test-time experience. Test-time training (TTT) addresses this limitation by enabling models to update their parameters on unlabeled test data, thereby extending their reasoning capabilities beyond the original training distribution. Recent methods such as EMPO 24 , TTRL 30 , and Theta-Evolve 18 employ self-generated reward signals such as entropy, majority voting, or self-consistency to refine reasoning policies via reinforcement learning without ground-truth labels. Practical implementations such as Cursor’s real-time reinforcement learning for Composer demonstrate the efficacy of test-time training in dynamically adapting model to complex, interactive coding environments 8 . Despite promising initial results, existing TTT methods for LRMs exhibit two fundamental limitations. First, they rely on heuristic reward signals that are intrinsically bounded by the model’s initial capabilities, leading to performance plateaus as self-improvement saturates 29 . Second, these methods tend to collapse output diversity in pursuit of higher average performance, ultimately degrading reasoning quality 25 . Both issues share a common root cause: there are no ground-truth labels available at test time, and the reward signal must be inferred from the model’s own outputs. As the model becomes increasingly confident in a narrow set of reasoning patterns, these heuristic signals systematically overestimate the quality of self-generated responses, creating a self-reinforcing loop that drives both saturation and diversity collapse. To this end, we propose Test-time Expectation-Maximization Policy Optimization ( TEMPO ), a TTT framework that decouples reward generation from policy optimization through an alternating actor-critic design (Figure 1 ). TEMPO operates in two stages: (i) Policy Refinement , where the actor generates reasoning trajectories on unlabeled test prompts and optimizes against rewards from a critic model, enabling on-the-fly adaptation to novel problems; and (ii) Critic Recalibration , where the critic is periodically updated using verifiable rewards from a labeled dataset. By maintaining a grounded critic, TEMPO provides a stable training signal that avoids the reward drift responsible for prior failures. We formalize this alternating procedure through the Expectation-Maximization (EM) algorithm. The key insight is that response correctness is an unobserved latent variable at test time. Thus, optimizing the policy without ground-truth labels is naturally framed as maximizing a lower bound on the expected reward. In this view, the critic recalibration corresponds to the E-step (estimating the posterior distribution over correct responses), while policy optimization corresponds to the M-step (updating model parameters given those estimates). This framing reveals that existing self-rewarding methods are degenerate variants that execute only the M-step, causing the estimated posterior to drift from true correctness. Restoring the E-step through periodic calibration tightens the lower bound and sustains improvement over extended training horizons. Experimental results validate both the effectiveness and scalability of TEMPO . On AIME 2024 and 2025, TEMPO improves OLMO3-7B from 33.0% and 26.3% to 51.1% and 37.0%. For Qwen3-14B, TEMPO pushes the accuracy from 42.3% and 37.1% to 65.8% and 44.6%, achieves an absolute gain of 23.5 and 7.5 percentage points, respectively. More importantly, TEMPO maintains high pass@K scores where baselines suffer from diversity collapse. Beyond mathematics, TEMPO generalizes to non-math reasoning domains, including logic puzzles and STEM tasks, confirming that the alternating critic-policy design is not domain-specific. Our contributions are summarized as follows: • We propose TEMPO , a test-time training framework that achieves sustained performance gains through an alternating actor-critic optimization, avoiding the diversity collapse and performance plateaus of prior LRMs self-training methods (Sec. 3 ). • We provide a unified analysis that characterizes existing TTT methods as incomplete EM procedures that omit the crucial posterior recalibration. By identifying the missing E-step as the root cause of scalability failures, this perspective yields a principled remedy: restoring periodic critic calibration on labeled data (Sec. 5 ). • We conduct extensive experiments across model families, scales (OLMO3-7B, Qwen3-8B, and Qwen3-14B), and five reasoning benchmarks spanning math, logic puzzles, and STEM. TEMPO demonstrates both superior accuracy and preserved output diversity, confirming that the alternating design generalizes beyond mathematical reasoning (Sec. 4 ). 2 Related Work This section surveys two lines of prior work that motivate our approach: self-rewarding RL methods that avoid ground-truth labels but suffer from reward self-reinforcement, and existing TTT methods for reasoning models that share the structural deficiency of omitting reward calibration. Self-rewarding reinforcement learning. RLVR, first formalized by Tulu-V3 9 , has emerged as the dominant paradigm for incentivizing reasoning capability in LLMs 7 ; 14 ; 4 , but its reliance on labeled data motivates self-rewarding alternatives. Previous works leverage intrinsic rewards such as entropy 24 ; 5 , self-certainty 26 , or reasoning topology 17 for self-training, without dependency on external supervision. For example, SARL 17 constructs rewards from the graph structure of intermediate reasoning steps, optimizing to encourage locally coherent and efficient thinking. LaSeR 20 demonstrates that the logit of the last token can serve as an effective self-rewarding signal, achieving superior reasoning accuracy and inference-time scaling with only one additional token of computation. However, these methods tend to collapse the output distribution and plateau as the reward signal becomes self-reinforcing 25 ; 29 . Our approach avoids this by decoupling reward generation (a critic periodically re-calibrated on labeled data) from

Read the complete text