CyclicReflex: Improving Large Reasoning Models via Cyclical Reflection Token Scheduling
Chongyu Fan Yihua Zhang Jinghan Jia Alfred Hero Sijia Liu · 2025
Read CyclicReflex: Improving Large Reasoning Models via Cyclical Reflection Token Scheduling by Chongyu Fan Yihua Zhang Jinghan Jia Alfred Hero Sijia Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11077.txt.
Introduction
Chongyu Fan and co-authors treat a reasoning model's moments of doubt as a resource to be scheduled. Their 2025 paper studies large reasoning models, which pause at special reflection tokens like wait or alternatively before continuing. Too much reflection wastes effort and can derail a solution; too little lets errors pass unchecked.
Borrowing from learning rate schedules in training, the team proposes CyclicReflex, a decoding rule that raises and lowers reflection token scores in a triangular wave along the answer. It needs no training and adds no computing cost. Across math contests, science questions, and coding benchmarks, and across model sizes, the method consistently beat standard decoding and recent rivals. The authors argue that when a model reflects matters as much as how much it reflects.
Catalog record
| CREATOR | Chongyu Fan Yihua Zhang Jinghan Jia Alfred Hero Sijia Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11077
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CyclicReflex: Improving Large Reasoning Models via Cyclical Reflection Token Scheduling Chongyu Fan Yihua Zhang Jinghan Jia Alfred Hero Sijia Liu Abstract Large reasoning models (LRMs), such as OpenAI’s o1 and DeepSeek-R1, harness test-time scaling to perform multi-step reasoning for complex problem-solving. This reasoning process, executed before producing final answers, is often guided by special juncture tokens or textual segments that prompt self-evaluative reflection. We refer to these transition markers and reflective cues as “ reflection tokens ” ( e.g. , “wait”, “but”, “alternatively”). In this work, we treat reflection tokens as a “ resource ” and introduce the problem of resource allocation , aimed at improving the test-time compute performance of LRMs by adaptively regulating the frequency and placement of reflection tokens. Through empirical analysis, we show that both excessive and insufficient use of reflection tokens, referred to as over-reflection and under-reflection, can degrade model performance. To better understand and manage this trade-off, we draw an analogy between reflection token usage and learning rate scheduling in optimization. Building on this insight, we propose cyclical reflection token scheduling (termed CyclicReflex), a decoding strategy that dynamically modulates reflection token logits using a position-dependent triangular waveform. Experiments on MATH500, AIME2024/2025, and AMC2023 demonstrate that CyclicReflex consistently improves performance across model sizes (1.5B–8B), outperforming standard decoding and more recent approaches such as TIP (thought switching penalty) and S1. Codes are available at https://github.com/OPTML-Group/CyclicReflex . 1 Introduction There has been a recent surge in the development of large reasoning models (LRMs), driven by the introduction of chain-of-thought (CoT) ( Wei et al. 2022 ) . Notable examples include OpenAI’s o1 ( OpenAI 2024 ) , Qwen 2.5 ( Yang et al. 2024 ) , DeepSeek-R1 ( Guo et al. 2025 ) , and Kimi-1.5 ( Team et al. 2025 ) . These models perform multi-step reasoning by generating so-called reflection tokens, phrases such as “wait”, “but”, “alternatively” , which signal hesitation, reconsideration, alternative exploration, or intermediate analysis. In parallel, test-time scaling techniques ( Snell et al. 2024 ; Liu et al. 2025 ) have emerged as a complementary approach for improving reasoning accuracy by expanding the breadth or depth of CoT traces during inference. However, LRMs remain prone to reasoning failures due to mismanagement of reflection tokens, often resulting in either underthinking or overthinking . Underthinking occurs when the model fails to fully explore promising reasoning paths for complex problems, often terminating prematurely or switching strategies too soon ( Wang et al. 2025a ; Su et al. 2025 ) . In contrast, overthinking arises when the model generates an excessive number of reflection tokens on simple problems, leading to unnecessary computational overhead ( Chen et al. 2024 ; Kumar et al. 2025a ) . These observations show that, as internal signals for deliberative reasoning, reflection tokens play a critical role in shaping answer quality. The emerging challenges further underscore the need for a principled mechanism to regulate reflection token usage during inference. In this work, we introduce the concept of resource allocation for LRMs , treating reflection tokens as a valuable resource whose scheduling along the CoT trajectory ( a.k.a. reasoning trace) can be strategically designed to improve reasoning accuracy. The objective is to determine the optimal usage, both in quantity and placement, of reflection tokens based on the difficulty of each problem. For instance, some problems exhibit under-reflection , where too few reflection tokens result in premature answer generation, while others suffer from over-reflection , where excessive tokens stall progress by repeatedly looping on phrases like “wait”. This raises a central question: (Q) How can we achieve effective resource allocation in LRMs to mitigate both under-reflection and over-reflection? To answer this, we draw a conceptual analogy between reflection tokens in LRMs and learning rates in optimization. Leveraging the landscape of thoughts ( Zhou et al. 2025 ) , we show that under-reflection mirrors the effect of an overly small learning rate, leading to premature convergence to suboptimal solutions, while over-reflection resembles a large learning rate that causes divergence. We briefly introduce our motivation and the underlying intuition below. Motivation: From stepsize hedging to cyclical learning rates. The critical role of learning rates (also known as stepsizes) in shaping optimization dynamics has been extensively studied ( Nesterov 1983 ; Allen-Zhu & Orecchia 2014 ; Bubeck et al. 2015 ) . A recent theoretical advancement, the silver stepsize schedule ( Altschuler & Parrilo 2024 ; Altschuler & Parrilo 2025 ) , demonstrates that replacing a constant learning rate with an approximately periodic, hedging-style schedule can provably accelerate convergence in gradient descent. This approach is known as stepsize hedging as it alternates strategically between large and small stepsizes–balancing rapid, but potentially unstable, exploration with slower, more stable convergence. A similar stepsize hedging idea has been applied to deep model training through cyclical learning rate schedule ( Smith 2017 ) , which alternates between large and small learning rates in a triangular waveform. This strategy not only accelerates convergence but also enhances generalization, often eliminating the need for extensive hyperparameter tuning. Figure 1: Schematic overview of our proposed method (CyclicReflex). The rightmost subfigure presents a comparison of final answer accuracy between CyclicReflex, the original LRM (DeepSeek-R1-Distill-Llama-8B), and decoding variants using TIP ( Wang et al. 2025a ) and S1 ( Muennighoff et al. 2025 ) . Motivated by the principle of stepsize hedging ( Altschuler & Parrilo 2024 ; Altschuler & Parrilo 2025 ) and the demonstrated effectiveness of cyclical learning rates in deep learning ( Smith 2017 ) , we propose CyclicReflex , a training-free decoding strategy that dynamically modulates the logits of reflection tokens using a position-dependent, periodic triangular waveform (see Fig. 1 ). Just as cyclical learning rates alternate between aggressive and conservative updates to balance exploration and convergence, CyclicReflex cyclically adjusts the sampling likelihood of reflection tokens to regulate the depth and stability of the reasoning process. Unlike conventional decoding methods, CyclicReflex is bidirectional , capable of both promoting and suppressing reflection token usage depending on the stage of generation. This flexibility enables CyclicReflex to address both under-reflection (insufficient reasoning) and over-reflection (excessive, redundant reasoning), offering a principled mechanism for reasoning modulation inspired by optimization dynamics. Contributions. We summarize our main contributions below. \bullet We introduce and formalize the novel problem of resource allocation in LRMs by treating reflection tokens as a computational resource, motivated by the dual challenges of under-reflection and over-reflection in reasoning generation. \bullet We draw a conceptual analogy between reflection token scheduling and learning rate scheduling in optimization, and validate it through the landscape of thoughts. Guided by that, we propose CyclicReflex, a test-time decoding strategy that cyclically modulates reflection token logits to dynamically balance reflection during generation. \bullet We conduct comprehensive experiments across four reasoning benchmarks and multiple model scales (1.5B–8B), demonstrating that CyclicReflex consistently improves both final-answer accuracy and self-correction capability, outperforming recent test-time decoding strategies such as TIP ( Wang et al. 2025a ) and S1 ( Muennighoff et al. 2025 ) . Moreover, CyclicReflex integrates seamlessly with other test-time scaling techniques, yielding additional performance gains. 2 Related Work LRMs and CoT. CoT ( Wei et al. 2022 ) enables LRMs to solve complex tasks through intermediate reasoning steps before reaching a final answer. This technique underpins