Token Level Routing Inference System for Edge Devices

Jianshu She, Wenhao Zheng, Zhengzhong Liu, Hongyi Wang, Eric Xing, Huaxiu Yao, Qirong Ho · 2025

Read Token Level Routing Inference System for Edge Devices by Jianshu She, Wenhao Zheng, Zhengzhong Liu, Hongyi Wang, Eric Xing, Huaxiu Yao, Qirong Ho (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-07878.txt.

Introduction

Jianshu She and six co-authors present a system that lets small language models on edge devices call on a large cloud model only when needed in a 2025 paper. Small models run fast and cheap but answer worse and invent facts more often. Large models answer better but are costly to run for every word.

Their system routes at the level of single tokens, the chunks that make up text. The small model writes most tokens on the device and sends only key ones to the cloud model. In a test on a laptop, a tiny model gained sixty percent in score while sending under seven percent of tokens to the cloud.

Catalog record

CREATORJianshu She, Wenhao Zheng, Zhengzhong Liu, Hongyi Wang, Eric Xing, Huaxiu Yao, Qirong Ho
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-07878

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Token Level Routing Inference System for Edge Devices Thanks: Demo package available at https://github.com/Jianshu1only/Token-Routing Jianshu She, Wenhao Zheng, Zhengzhong Liu, Hongyi Wang, Eric Xing, Huaxiu Yao, Qirong Ho Abstract The computational complexity of large language model (LLM) inference significantly constrains their deployment efficiency on edge devices. In contrast, small language models offer faster decoding and lower resource consumption but often suffer from degraded response quality and heightened susceptibility to hallucinations. To address this trade-off, collaborative decoding, in which a large model assists in generating critical tokens, has emerged as a promising solution. This paradigm leverages the strengths of both model types by enabling high-quality inference through selective intervention of the large model, while maintaining the speed and efficiency of the smaller model. In this work, we present a novel collaborative decoding inference system that allows small models to perform on-device inference while selectively consulting a cloud-based large model for critical token generation. Remarkably, the system achieves a 60% performance gain on CommonsenseQA using only a 0.5B model on an M1 MacBook, with under 7% of tokens generation uploaded to the large model in the cloud. 1 Introduction Figure 1: System overview: First transfer Huggingface model to ONNX model, then add hidden states of last layer as a output node in ONNX computation graph, deploy ONNX model on Laptop and ONNX-mobile on Mobile phone. Then connect edge divice with router to the SG-Lang backend from server side. The router automatically route token with low confidence to server, and send response back to edge device Large language models (LLMs) have transformed natural language processing, achieving state-of-the-art performance in tasks such as document summarization, question answering, and text generation. Models like Meta’s Llama series Touvron et al. 2023 , Google’s Gemma Team et al. 2024 , and DeepSeek series DeepSeek-AI et al. 2025 have demonstrated remarkable capabilities, driving advancements in various applications. However, their deployment in edge devices, such as smartphones, embedded systems, and Internet of Things (IoT) devices, faces significant hurdles due to their high computational complexity ( Zhang et al. 2024a ; Lin et al. 2024 ) . The role of small language models (SLMs), and the emerging paradigm of collaborative decoding, culminating in a novel framework that balances efficiency and performance. The computational demands of LLMs, such as the Llama-2 7B parameter model requiring over 8GB of memory in FP16 precision Zhang et al. 2024a , exceed the capabilities of many edge devices, like the NVIDIA Jetson Orin Nano with 8GB DRAM Shen et al. 2024a ; Li et al. 2025 . This limitation is compounded by hardware heterogeneity, including ARM processors in smartphones and low-power IoT chips, which further complicates deployment ( Dao et al. 2022 ) . Recent works, such as Zheng et al. 2025b , highlight the need for solutions that can operate within the constraints of memory, processing power, and energy consumption ( Miao et al. 2024 ) . One promising approach to leveraging small language models (SLMs) lies in their potential for edge deployment, thanks to their reduced size and faster inference times ( Xue et al. 2024 ; Jiang et al. 2023 ; Zhou et al. 2024 ) . These models consume fewer resources, making them suitable for devices with limited capabilities. However, studies, such as Wang et al.’s work on large and small model trade-offs Zheng et al. 2025b , indicate that SLMs often suffer from degraded response quality and increased susceptibility to hallucinations—generating factually incorrect content ( Xu et al. 2023 ) . This trade-off between efficiency and performance presents a critical barrier, particularly for applications requiring high accuracy, such as medical data analysis or financial processing ( Wang et al. 2024 ) . To mitigate this trade-off, numerous studies have introduced approaches that dynamically route input queries to models of varying sizes, aiming to lower inference costs without compromising output quality ( Kou et al. 2024 ; Anagnostidis et al. 2024 ) . Collaborative decoding has emerged as a promising approach ( Shen et al. 2024b ; Shi et al. 2024 ) . This paradigm involves SLMs handling the bulk of the inference process while LLMs assist in generating critical tokens, such as those with high uncertainty or decisive impact on the output. Research suggests that this method leverages the strengths of both model types, maintaining efficiency while enhancing quality. For instance, Wang et al.’s study on Fast and Slow Generating (FS-GEN) Zhang et al. 2024b categorizes LLMs as System 2 (slow and deliberate) and SLMs as System 1 (fast and intuitive), finding that collaborative interactions require less than 20% of the computations, following scaling laws based on parameter ratios. Building on these insights, we introduce a novel token-level routing inference system for edge devices, addressing the challenge of balancing efficiency and performance in resource-constrained settings. The system enables on-device SLMs to perform primary decoding while selectively routing critical tokens to a cloud-based LLM using a lightweight, confidence-based MLP router (See Figure 1 for details). Empirical results on CommonsenseQA demonstrate that routing only 7% of tokens to the LLM yields over 60% accuracy improvement, with more than 80% cost reduction compared to full LLM inference. This system paves the way for practical, low-latency, high-quality language model applications on edge hardware, as it mitigates the traditional trade-off between model size and performance, opening new possibilities for deploying high-quality language models in resource-constrained environments. For example, in privacy-sensitive scenarios like medical data analysis, on-device inference reduces data transmission, protecting user data, while cloud-based LLM assistance ensures accuracy. Unlike prior works which focus solely on routing algorithms, our contribution lies in building a fully operational client-server token routing system compatible with edge deployment. This includes integration with ONNX inference on laptops and phones, low-latency LLM serving, and practical routing logic—bringing theoretical ideas into real-world applications. 2 Token Level Routing Figure 2: Computation procedure: Unlike conventional inference, the token routing system involves multiple rounds of prefill and decode within a single request, which prevents full utilization of inference acceleration engines such as SGLang and vLLM, as they only optimize kernel and KV cache on single stage prefill and decode. In this section, we introduce serveral token level routing algorithm that can be used on our system. 2.1 CITER – Collaborative Inference with Token-level Routing CITER ( Zheng et al. 2025a ) is a framework that accelerates language model inference through token-level routing between a small, fast but less accurate language model (SLM) and a large, accurate but expensive model (LLM). A trainable router determines, for each token, whether to use the SLM or the LLM, based on routing scores and a predefined threshold \tau . To capture the long-term tradeoff between cost and quality, CITER formulates router training as a preference-based reinforcement learning problem over a Markov Decision Process (MDP). Each state consists of the input prompt and the current generated tokens, and the actions correspond to choosing either the SLM or LLM to generate the next token. Rewards reflect both inference efficiency and the quality of the final generated response. Rather than specifying explicit reward functions, CITER leverages pairwise routing preferences: whether generating a token with the SLM is preferred over the LLM. These preferences are modeled using the Bradley-Terry model and optimized via

Read the complete text