RelayLLM : Efficient Reasoning via Collaborative Decoding

Chengsong Huang, Tong Zheng, Langlin Huang, Jinyuan Li Haolin Liu Jiaxin Huang, [0.5ex] Washington University in St. Louis · 2026

Read RelayLLM : Efficient Reasoning via Collaborative Decoding by Chengsong Huang, Tong Zheng, Langlin Huang, Jinyuan Li Haolin Liu Jiaxin Huang, [0.5ex] Washington University in St. Louis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05167.txt.

Introduction

Chengsong Huang and 5 co-authors let small and large language models share a single answer in a 2026 paper. Complex reasoning with a large model is costly and slow, while small models are cheap but often lack reasoning depth. Existing collaboration routes whole queries to the large model through cascading or routing, wasting money whenever the small model could have handled most of the reasoning steps alone.

Their framework, RelayLLM, collaborates at token level: the small model drafts, and the large model steps in only where needed during decoding. Token-level handover trims the waste of query-level routing. Cheaper reasoning matters for serving assistants at scale, where most tokens are routine and only a few hard steps truly need the biggest model's attention.

Catalog record

CREATORChengsong Huang, Tong Zheng, Langlin Huang, Jinyuan Li Haolin Liu Jiaxin Huang, [0.5ex] Washington University in St. Louis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05167

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RelayLLM : Efficient Reasoning via Collaborative Decoding Chengsong Huang, Tong Zheng, Langlin Huang, Jinyuan Li Haolin Liu Jiaxin Huang, [0.5ex] Washington University in St. Louis Abstract Deploying Large Language Models (LLMs) for complex reasoning is often hindered by high computational costs and latency, while resource-efficient Small Language Models (SLMs) typically lack the necessary reasoning capacity. Existing collaborative approaches, such as cascading or routing, operate at a coarse granularity by offloading entire queries to LLMs, resulting in significant computational waste when the SLM is capable of handling the majority of reasoning steps. To address this, we propose RelayLLM , a novel framework for efficient reasoning via token-level collaborative decoding. Unlike routers, RelayLLM empowers the SLM to act as an active controller that dynamically invokes the LLM only for critical tokens via a special command, effectively "relaying" the generation process. We introduce a two-stage training framework, including warm-up and Group Relative Policy Optimization (GRPO) to teach the model to balance independence with strategic help-seeking. Empirical results across six benchmarks demonstrate that RelayLLM improves the average accuracy from 42.5% to 49.52%, effectively bridging the performance gap between the two models. Notably, this is achieved by invoking the LLM for only 1.07% of the total generated tokens, offering a 98.2% cost reduction compared to performance-matched random routers. Our code is available at https://github.com/Chengsong-Huang/RelayLLM . 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning and problem-solving ( 4 ; 49 ; 1 ) . However, their deployment is often constrained by high computational costs and latency. In contrast, Small Language Models (SLM) are resource-efficient options but typically struggle with hard reasoning tasks due to their limited capacity ( 21 ) . This trade-off has motivated the development of collaborative systems that combine the reasoning capabilities of LLMs with the efficiency of smaller models ( 12 ) . Existing approaches to different-sized model collaboration often rely on “cascading” or “routing” mechanisms, where a router determines the difficulty of a query and directs it to either a small or large model ( 6 ; 15 ; 33 ) . While effective to some extent, these methods typically operate at a coarse granularity by offloading the entire generation task to the large model once a query seems difficult. This “all-or-nothing” strategy leads to significant computational waste, as the small model often possesses the competence to handle the majority of the reasoning steps, requiring expert intervention only at specific critical positions ( 29 ; 35 ) . Figure 1: Results are averaged across six mathematical benchmarks. The “Random Router” baseline randomly directs questions to either the small or large model. The “Perfect Router” baseline directs only questions SLM cannot solve to large model. The x-axis represents the Call Ratio (percentage of tokens generated by the teacher model), and the y-axis denotes the average accuracy. To address these problems, we propose RelayLLM , a novel framework for efficient reasoning via token-level collaborative decoding ( 37 ) without an additional controller. Unlike static routers, RelayLLM empowers the small model to act as both a problem solver and an active controller that dynamically requests assistance only when necessary. Inspired by tool-use agents ( 47 ; 56 ) , we introduce an interleaved generation process where the small model generates a special command token ( <call> ) to pause its own generation and invoke the large model for a specified number of tokens. The small model then receives the expert’s guidance and resumes reasoning, effectively “relaying” the output process between models. We propose a two-stage framework to equip the small model with this strategic delegation capability to train the RelayLLM . We first employ a supervised warm-up phase to teach the model the syntactic structure of calling commands. This is followed by a reinforcement learning stage using Group Relative Policy Optimization (GRPO) ( 36 ) training, where we design a context-aware reward that guides the model to balance independence with necessary help-seeking, penalizing both wasted costs and avoidable errors. Empirical results on six benchmarks demonstrate the effectiveness of our approach. As illustrated in Figure 1 , RelayLLM achieves an average accuracy of 49.52% on six benchmarks, largely recovering the performance gap between the small model and the large one. Remarkably, this gain is achieved with minimal cost, as RelayLLM invokes the large model for only 1.07% of the total generated tokens. Furthermore, in comparison to a resource-equivalent Random Router, RelayLLM yields a substantial 6.9% accuracy improvement. These results confirm that RelayLLM effectively identifies critical reasoning steps for expert intervention, reducing token costs by 98.2% compared to a performance-matched router. Surprisingly, evaluations in a teacher-free setting reveal that the model internalizes effective reasoning patterns during collaboration, enabling it to surpass baselines on easier benchmarks even without expert assistance. 2 RelayLLM Inference Figure 2: Overview of the RelayLLM framework. (Left) Collaborative Inference: The Small Language Model acts as a central controller. During generation, it can actively trigger an “intervention” by generating a <call> command. The Large Language Model then generates the specified number of tokens, after which control returns to the SLM to complete the reasoning. (Right) Difficulty-Aware Reward Design: During GRPO training, we sample a group of rollouts (both with and without LLM intervention) and classify the query difficulty into three distinct scenarios: Solvable , Teacher-Dependent , and Unsolvable , to guide the policy optimization.Distinct reward designs are applied to each scenario to align the model’s behavior with the optimal strategy. As illustrated in Fig. 2 , we consider a hybrid inference setting involving a primary, resource-efficient Small Language Model (SLM), denoted as \mathcal{M}_{S} , and a powerful but computationally expensive Large Language Model (LLM), denoted as \mathcal{M}_{L} . Given an input query \mathbf{x} , the system aims to generate a high-quality response \mathbf{y} through dynamic collaboration. Different from standard cascading methods that simply offload the entire remaining task to a larger model, we define an interleaved generation process where the small model acts as both reasoner and central controller. The process operates as follows: 2.1 Small Model Generation By default, \mathcal{M}_{S} generates tokens autoregressively based on the current context history as a normal language model. \mathcal{M}_{S} is augmented with a special control capability: it can actively request assistance by generating a specific command pattern: \mathcal{C}_{\text{cmd}}(n)=\texttt{<call>}\oplus n\oplus\texttt{</call>} where \oplus denotes string concatenation, and n\in\mathbb{Z}^{+} represents the number of tokens required from the larger language model. 2.2 Large Model Intervention When this trigger pattern is detected, the generation by \mathcal{M}_{S} pauses. The current context (including query and tokens generated by SLM) is forwarded to \mathcal{M}_{L} . Crucially, to maintain compatibility with the large model’s standard input distribution, we strip the special command tokens ( \mathcal{C}_{\text{cmd}} ) from the context provided to \mathcal{M}_{L} . The large model then generates the next n tokens (or stops early if an end-of-sequence “[EOS]” token is reached), providing a high-quality continuation or reasoning step. 2.3 Iterative Relay of LLM and SLM After \mathcal{M}_{L} completes the generation, control is returned to the small model. The context is updated with the new tokens generated by \mathcal{M}_{L} added into the existing context. Unlike the large model, \mathcal{M}_{S} retains the full history, including its own generated command tokens (<call>n</call>). This allows \mathcal{M}_{S} to maintain a trace of its active delegation

Read the complete text