SplitReason: Learning To Offload Reasoning

Yash Akhauri, Anthony Fei, Chi - Chih Chang, Yueying Li, Mohamed S. Abdelfattah · 2025

Read SplitReason: Learning To Offload Reasoning by Yash Akhauri, Anthony Fei, Chi - Chih Chang, Yueying Li, Mohamed S. Abdelfattah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16379.txt.

Introduction

Yash Akhauri and co-authors teach a small reasoning model to hand off its hardest steps to a bigger one. Long chains of thought make language models accurate but slow and costly, since every token is produced one by one. Not all steps are equally hard, though, so the team built SplitReason to keep the easy stretches in-house and offload only the tough segments.

The authors marked difficult segments in eighteen thousand reasoning traces from a public math dataset, then trained a 1.5-billion-parameter model with supervised and reinforcement fine-tuning to spot and delegate such segments itself. On a hard math benchmark the setup raised accuracy by twenty-four to twenty-eight percent while offloading only about one to five percent of tokens. Model, data, code, and logs are released openly.

Catalog record

CREATORYash Akhauri, Anthony Fei, Chi - Chih Chang, Yueying Li, Mohamed S. Abdelfattah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16379

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SplitReason: Learning To Offload Reasoning Yash Akhauri, Anthony Fei, Chi - Chih Chang, Yueying Li, Mohamed S. Abdelfattah Abstract Reasoning in large language models (LLMs) tends to produce substantially longer token generation sequences than simpler language modeling tasks. This extended generation length reflects the multi-step, compositional nature of reasoning and is often correlated with higher solution accuracy. From an efficiency perspective, longer token generation exacerbates the inherently sequential and memory-bound decoding phase of LLMs. However, not all parts of this expensive reasoning process are equally difficult to generate. We leverage this observation by offloading only the most challenging parts of the reasoning process to a larger, more capable model, while performing most of the generation with a smaller, more efficient model; furthermore, we teach the smaller model to identify these difficult segments and independently trigger offloading when needed. To enable this behavior, we annotate difficult segments across 18k reasoning traces from the OpenR1-Math-220k chain-of-thought (CoT) dataset. We then apply supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) to a 1.5B-parameter reasoning model, training it to learn to offload the most challenging parts of its own reasoning process to a larger model. This approach improves AIME24 reasoning accuracy by 24% and 28.3% while offloading 1.35% and 5% of the generated tokens respectively. We open-source our SplitReason model , data-set , code and logs . Figure 1: SplitReason intelligently offloads token generation to a large model during difficult parts of the reasoning process. Leveraging a small model (1.5B parameters) for majority of the decode process leads to significant end-to-end speedup compared to the large model (32B parameters), while improving accuracy over the small model. 1 Introduction Large language models (LLMs) are powerful general-purpose learners that excel at a wide range of tasks [ 1 , 2 , 3 ] . Recent advances in LLM post-training have shown that their performance on reasoning-heavy tasks can be improved by inducing the ability to reason by generating explicit chain-of-thoughts (CoT) about a question before arriving at the final answer [ 4 ] . However, this shift towards more complex, multi-step reasoning during inference [ 5 ] significantly increases test-time compute cost. In practice, LLMs often have to generate thousands of tokens while referencing all previously produced tokens via a Key-Value Cache (KV-Cache) for every new token. This process is memory-bound and grows quadratically with respect to sequence length [ 6 , 7 , 8 ] , making it very time-consuming as we scale up model sizes and rely on longer CoT to improve reasoning [ 9 , 10 ] In addition, further increasing compute at test time improves accuracy on reasoning tasks such as AIME24 and MATH500 [ 11 ] . This leads to an explosion in the thinking time needed: thousands of tokens are used for CoT reasoning before generating the final answer. Figure 2: SplitReason utilizes two models to perform fast and high-accuracy reasoning. A small model is fine-tuned to emit a <bigmodel> tag when it detects a difficult reasoning step. This triggers a large model to step in and take over generation until a </bigmodel> tag is detected. We hypothesize that reasoning segments are not uniformly difficult—certain parts of a problem can be generated with less effort using a small model, while others require more complex reasoning using larger models. Figure 2 illustrates an example of our approach: SplitReason . A small LLM (e.g., 1.5B parameters) begins generation by processing the question and starting to think about the problem, generating an initial CoT as it reasons through the steps. When the small model encounters a difficult part of the reasoning process, it independently emits a <bigmodel> token to request a reasoning segment from a much larger model (e.g., 32B parameters). In parallel, the large model batch-processes (prefill mode) the small model’s output, enabling immediate continuation of generation whenever offloading is triggered. The roles are then reversed: the large model generates a CoT segment for the difficult part of the reasoning process (in decode mode), while the small model performs prefill on the large model’s output to immediately continue generation, or to check if the small model wants to take back control by emitting a </bigmodel> token. This process can repeat multiple times until the final answer is produced. Crucially, the most expensive part of generation—decode mode on the large model—is minimized, with the additional cost of prefill computations on both the small and large models. A 1.5B/32B model can perform prefill and decode at \sim 30,000/2,500 and \sim 150/15 tokens/s respectively 1 1 1 Measured on A6000 GPUs on vLLM v0.8.3 for Qwen models. Two GPUs are used for models larger than 8B. , highlighting a massive speed difference between large model decode and everything else. A key challenge is determining when to switch between the small and large models based on reasoning difficulty, specifically, how to train the small model to emit the <bigmodel> and </bigmodel> tokens at the appropriate points in the reasoning process? To achieve this, we annotate a CoT dataset (OpenR1-Math-220k) with easy and difficult segments, delimited by the special <bigmodel> and </bigmodel> tokens. While this annotation could be performed manually, it is far more efficient and scalable to leverage a high-quality LLM for this task; we opt for the DeepSeek-R1 671B model. The resulting annotated dataset is then used to perform supervised fine-tuning (SFT) on the small model, training it to insert the special tokens at appropriate points in the reasoning process. Finally, we apply reinforcement learning fine-tuning (RLFT) to regulate and encourage the emission of the <bigmodel> token, balancing downstream task accuracy against overall generation latency. Our methodology is general and can be applied to different model efficiency approaches beyond SplitReason. Generally, R einforcement L earning for optimizing E fficiency ( RL4E ) introduces a new paradigm by which we use fine-tuning to enable LLMs to become inherently more efficient by including measures of hardware efficiency during fine-tuning. We enumerate our contributions below: • We develop and open-source a fine-tuning dataset and recipe, to enable models to learn when to offload their own reasoning process to a larger model. • We demonstrate that accuracy of small reasoning models can be improved by 28.3% by offloading \sim 5% of the reasoning process to larger models. This can speed up inference by 4-6\times . • We show that models can learn when a task is difficult, and can leverage RL4E to attain higher efficiency. This enables a new paradigm in which models are taught to align not just with human preferences, but with hardware preferences too. 2 Background Test-Time Scaling: Early work on prompting showed that pretrained LLMs can reason if provided explicit CoT instructions in the prompt [ 10 ] . However, this method is brittle and has a large inference-time token budget requirement. A more robust method to induce reasoning is with Supervised Fine-Tuning (SFT) on high-quality CoTs. The model is shown questions formatted with <think> CoT </think> answer, which teaches the model to imitate the reasoning trajectory. SFT has been used to induce an internal <think> stage that can be exploited at test time [ 12 ] . However, SFT is fundamentally an imitation procedure, where the policy is rewarded for matching every token in

Read the complete text