CIRF: Tokenizing Chain-of-Thoughts into Reusable Functional Units for Efficient Latent Reasoning in Large Language Models
Yukyung Lee Yumeng Shen Jinhyeong Park Hyein Yang Jun-Hyung Park · 2026
Read CIRF: Tokenizing Chain-of-Thoughts into Reusable Functional Units for Efficient Latent Reasoning in Large Language Models by Yukyung Lee Yumeng Shen Jinhyeong Park Hyein Yang Jun-Hyung Park (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28292.txt.
Introduction
Lee and colleagues shrink chain-of-thought reasoning into reuse-ready functional tokens for cheaper hidden inference. CIRF gives one job token to each clear reasoning chunk in plain traces, then trains models to emit tokens and optional results before answering. Hidden reasoning thus tracks plain rationales, trading correctness against wait time well.
Catalog record
| CREATOR | Yukyung Lee Yumeng Shen Jinhyeong Park Hyein Yang Jun-Hyung Park |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56555 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28292
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CIRF: Tokenizing Chain-of-Thoughts into Reusable Functional Units for Efficient Latent Reasoning in Large Language Models Yukyung Lee Yumeng Shen Jinhyeong Park Hyein Yang Jun-Hyung Park Abstract Implicit Chain-of-Thought (CoT) reduces the inference cost of large language models by internalizing the explicit rationales. However, existing approaches typically lack alignment with explicit rationales and adaptivity to example complexity. In this work, we propose CIRF ( C hain-of-thoughts I nto R eusable F unctional units ), an implicit CoT framework that performs reasoning as a dynamic sequence of discrete functional tokens. CIRF assigns a functional token to each semantically coherent reasoning unit in explicit CoT traces. The model is then fine-tuned to autoregressively generate functional tokens and their optional results, followed by the final answer. This design aligns latent reasoning with a sequence of functional units, facilitating parallel training, explicit rationale alignment, and adaptive reasoning. Extensive experiments on mathematical, symbolic, and commonsense reasoning benchmarks show that CIRF provides a favorable accuracy-latency trade-off compared with state-of-the-art implicit CoT methods. Further analyses demonstrate that CIRF constructs distinct, interpretable functional tokens, leading to consistent performance improvements. Figure 1: Overview of CIRF. Each functional unit in a CoT rationale is encoded using a sentence embedding model, mean-centered to reduce question-specific situational bias, and quantized into a discrete functional token. The target LLM is then fine-tuned to generate a compact sequence consisting of functional tokens, optional result tokens, and the final answer tokens. CIRF enables efficient latent reasoning through discrete functional tokens. 1 Introduction Chain-of-thought (CoT) reasoning 26 ; 25 has become a widely used strategy for eliciting multi-step reasoning in large language models (LLMs). By prompting or training a model to generate intermediate rationales before producing the final answer, CoT has substantially improved performance on mathematical and symbolic reasoning tasks. Recent analysis 19 suggests that the gains from CoT are strongest in domains that require symbolic or multi-step computation, further highlighting the importance of traceable intermediate steps for solving complex problems. Despite its effectiveness, the latency and memory costs incurred by long reasoning traces have motivated research on implicit CoT 2 ; 7 . Instead of generating full rationales in natural language, these studies aim to internalize, compress, or replace explicit reasoning traces using hidden states or special tokens. They have shown that CoT can be performed implicitly within models, thereby reducing the cost of generating additional natural language tokens. A key requirement for successful internalization is to align the LLM states during implicit CoT with those induced by explicit CoT. Typical implicit CoT methods utilize repeated placeholder tokens 6 ; 31 for alignment. However, these methods require the model to organize long, dynamic reasoning traces over a sequence whose surface form is largely homogeneous, whereas typical autoregressive LLMs are trained to reason conditioned on growing, changing textual prefixes. Whether such identical carriers provide an effective substrate for multi-step reasoning remains underexplored. To address this issue, several methods 7 ; 18 autoregressively feed output hidden representations as the next token embeddings, thereby reflecting changes in input token states as reasoning proceeds. However, training with recursively fed hidden representations incurs substantial training cost and instability, particularly in modeling multiple implicit tokens. In their formulations, input representations at each step need to be sequentially generated in training, because they are unknown until the model finishes computation at the previous step. Furthermore, their training procedures typically follow long curriculum stages or require a teacher model. The difficulty in scaling the number of implicit tokens may limit performance, considering that recent work 8 increasingly shows that the number of reasoning tokens is important: too few tokens can lead to underthinking, while too many can cause overthinking, instability, or wasted computation. We propose a new implicit CoT framework, called CIRF ( C hain-of-thoughts I nto R eusable F unctional units ), which represents reasoning as an autoregressively generated sequence of discrete functional tokens. The key idea is to tokenize explicit CoT traces into functional units. Consequently, each rationale is converted into a short, dynamic sequence of function-bearing tokens, enabling parallel training over sequences whose lengths adapt to the complexity of the reasoning process. We then fine-tune a single universal LLM to generate these implicit functional tokens, followed by optional results and the final answer for alignment, across diverse reasoning datasets. This design addresses the above issues by utilizing semantically grounded and adaptively allocated functional tokens. Through extensive experiments, we empirically verify that CIRF is more efficient than state-of-the-art implicit CoT methods across diverse reasoning benchmarks. We evaluate CIRF in terms of accuracy-latency trade-off with diverse configurations partially mixing implicit and explicit tokens. The results show that CIRF achieves the accuracy–latency frontier. In-depth analyses further reveal that CIRF provides interpretable functional token assignment and adaptive token sequences, leading to substantial performance improvements. 2 Related Work 2.1 Explicit CoT It has been observed that LLMs can generate intermediate reasoning steps that improve the accuracy of final answers 26 . Zero-shot CoT 13 has shown that simple verbal triggers can elicit step-by-step reasoning without manually designed exemplars. While these findings highlight the importance of traceable intermediate steps for solving complex problems, many studies 8 ; 28 have also reported an inverted-U relationship between CoT length and accuracy: reasoning traces that are either too short or too long can hurt performance. Several studies 30 ; 29 have proposed methods for identifying shorter rationales that can match or exceed verbose CoT sequences. These studies are related to CIRF in that they show both the efficacy of generating intermediate reasoning steps and the importance of reasoning length for accuracy and efficiency. 2.2 Implicit CoT Implicit CoT approaches aim to internalize reasoning into model hidden states. Early approaches have trained the hidden states associated with answer tokens to mimic the hidden states of explicit CoT 2 . Placeholder tokens 6 ; 16 have also been used as learnable thinking symbols. However, these methods may limit the reasoning capabilities of LLMs due to the lack of semantically grounded tokens. Another line of work has conducted reasoning through recursively fed output hidden states 7 ; 18 . However, these approaches have introduced substantial complexity in training and offered limited adaptivity in reasoning length. A further subgroup of research has used auxiliary models to generate compact reasoning surrogates 31 ; 9 ; 27 , causing additional cost from the auxiliary models. Compared with these methods, CIRF provides a more efficient, straightforward strategy via a sequence of discrete functional tokens: facilitating single stage training with teacher forcing, realizing fine-grained reasoning functions and inducing adaptive reasoning behavior in a single unified model. 2.3 Tokenization of Latent Representations 24 have explicitly inferred planning variables using clustering over step representations, and its probing analysis has used pre-trained sentence encoders to study whether the induced planning categories are learnable and distinct. It has clustered reasoning steps to create auxiliary labels that precede explicit CoT. 20 have used generated latent discrete tokens to randomly replace the initial portion of reasoning traces, showing that a discrete latent vocabulary can be learned. Compared with these methods, CIRF fully segments explicit CoT into semantically grounded units and introduces a carefully designed alignment method, leading to substantial improvements in reasoning efficiency. 3 Methodology The proposed CIRF framework converts an explicit CoT rationale