OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decoding
Ramchalam Kinattinkara Ramakrishnan , 1 Zhaocong Yuan , 2 · 2025
Read OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decoding by Ramchalam Kinattinkara Ramakrishnan , 1 Zhaocong Yuan , 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02659.txt.
Introduction
Ramchalam Ramakrishnan and co-authors presented OmniDraft in 2025, a draft model that speeds up language model answers on the device itself. Speculative decoding lets a small model propose tokens that a big model checks, but the drafter usually must be built for one target family. In live service two problems follow: a swapped target breaks the match, and speed should improve as usage grows.
One drafter here works with any target. An online store of word sequences plus mixed distillation training bridges the vocabulary gap between drafter and target, and adaptive drafting tunes the pace over time. The catalog creator field garbles the list; the head byline, which runs to seven names, governs. Readers get a speed layer designed for phones and other local hardware, where one drafter must serve many masters.
Catalog record
| CREATOR | Ramchalam Kinattinkara Ramakrishnan , 1 Zhaocong Yuan , 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-02659
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decoding Thanks: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Ramchalam Kinattinkara Ramakrishnan , 1 Zhaocong Yuan , 2 Abstract Speculative decoding generally dictates having a small, efficient draft model that is either pretrained or distilled offline to a particular target model series, for instance, Llama or Qwen models. However, within online deployment settings, there are two major challenges: 1) usage of a target model that is incompatible with the draft model; 2) expectation of latency improvements over usage and time. In this work, we propose OmniDraft, a unified framework that enables a single draft model to operate with any target model and adapt dynamically to user data. We introduce an online n-gram cache with hybrid distillation fine-tuning to address the cross-vocabulary mismatch across draft and target models; and further improve decoding speed by leveraging adaptive drafting techniques. OmniDraft is particularly suitable for on-device LLM applications where model cost, efficiency and user customization are the major points of contention. This further highlights the need to tackle the above challenges and motivates the “one drafter for all” paradigm. We showcase the proficiency of the OmniDraft framework by performing online learning on math reasoning, coding and text generation tasks. Notably, OmniDraft enables a single Llama-68M model to pair with various target models including Vicuna-7B, Qwen2-7B and Llama3-8B models for speculative decoding; and additionally provides up to 1.5-2x speedup. 1 Introduction Unlike traditional auto-regressive generation in LLMs, speculative decoding (SpD) leviathan2023fast ; chen2023accelerating offers a unique advantage to accelerate LLM inference by decoupling the generation phase and verification phase. Speculative decoding generally requires a small but efficient draft model and a large target model. The draft model generates a sequence of proposed tokens to be verified by the target in one shot, amortizing the target model’s memory bottleneck in batch inference and attaining better tokens per second throughput. The speedup factor relies not only on the predictability of the generated text like commonly occurring phrases, but also on the alignment between draft and target model. As such, a common practice is to utilize draft and target models from the same model family given their consistency in pretrained data, tokenization and training configurations. Alternatively, one might consider distilling a target model into a smaller model to serve as the drafter zhoudistillspec ; liuonline , which still follows the same principle of better alignment leading to greater speedup. The tight coupling of draft and target models limits flexibility of model selection and creates additional overhead for draft model distillation and maintenance, especially when deploying LLMs at scale across diverse hardware platforms and updating target models overtime. It is compelling to use a universal lightweight model on-device to draft tokens for a broad range of targets models. This universality greatly simplifies deployment, facilitates easy optimization, and allows for rapid model updates. Furthermore, the heterogeneity of on-device and cloud hardware presents an opportunity for hybrid speculative decoding. This enables users to choose between running any target model locally or on the cloud, balancing model performance, inference cost, and privacy concerns. However, building a universal drafter for speculative decoding presents several unique challenges. Firstly, different family of target models might use tokenizers with different vocabularies. This is natural since target models are typically trained with massive pretrained data and hence a larger vocabulary is needed to include higher-order n-grams or BPE bostrom2020byte merges. As a result, the vocabulary mismatch breaks the speculative decoding formulation where the draft and target model need to evaluate probabilities over the same set of tokens. A previous work UAG timor2025acceleratingllminferencelossless addresses vocabulary mismatch with an intermediate translation layer, but it primarily deals with tokens within the intersection of the drafter and target vocabularies, which might "falsely" reject good tokens from the drafter. Secondly, independent training of the drafter and target models can result in misaligned predicted token distributions, which reduces the acceptance rate during verification. This misalignment diminishes the efficiency benefits of speculative decoding. Although existing research zhoudistillspec ; liuonline ; minixhofer2025cross has explored techniques to improve alignment between the drafter and target, the alignment is often done offline with assumption of a fixed target model. In practice, the target model may change due to user personalization or tasks switching, further complicating the alignment issue. Additionally, edge devices usually have limited memory, compute capacity, and power budget. The drafting process must therefore be efficient to maximize the benefits of speculative decoding. Figure 1: Overview of the OmniDraft framework: during cross-vocabulary speculative decoding, the drafter (Llama-68M) generates multiple tokens d_{i} with corresponding distributions q_{i} . Cross-vocabulary translator then converts the drafter tokens into tokens in the vocabulary of the target model (Llama3-8B). In this example, token d_{0} (’Snow’) and d_{4} (’is’) are directly mapped to target tokens t_{0} and t_{2} , while token d_{1} (’f’), d_{2} (’la’) and d_{3} (’ke’) are merged into a single target token t_{1} (’flake’), since there is a mapping item in the n-gram cache. The translated proposal t_{i} along with combined probabilities q^{\prime}_{i} is verified by the target model, resulting in t_{0} and t_{1} being accepted while t_{2} being rejected and replaced by t^{\prime}_{2} . The target outputs tokens and their probabilities p_{i} are translated into drafter tokens and sent back to drafter for next round of drafting. The n-gram cache is updated by inserting a new unseen item (’st’,’amps’->’stamps’). Meanwhile, the accepted and corrected tokens from the target model are used to align the drafter through online cross-vocabulary distillation. To address the challenges, we propose a scalable speculative decoding framework, named OmniDraft , centered on an on-device universal drafter that generates draft tokens for a wide variety of target models. An overview of OmniDraft is shown in Figure 1 . To tackle the vocabulary mismatch issue and enable cross-vocabulary speculative decoding, we introduce an n-gram cache to store cross-vocabulary mappings from draft tokens to target tokens. By integrating the n-gram cache into the speculative decoding algorithm, we alleviate vocabulary mismatches and achieve a higher acceptance rate for future queries. We further employ online knowledge distillation to improve alignment between the drafter and target. A hybrid distillation loss, combining token-level and distribution level objectives, updates the drafter using the target’s accepted and corrected outputs. This enables continuous alignment during speculative decoding, even when the target model changes due to personalization or task switching. To further improve runtime efficiency, we incorporate online adaptive drafting, where the drafter dynamically adjust the number of tokens it proposes based on predicted confidence. The adaptive drafting balances generation cost and acceptation rate, maximizing throughput under device constraints. Overall, our OmniDraft framework enables a robust, efficient and flexible speculative decoding system with a universal drafter for on-device applications. Through extensive experiments, we show that a single Llama-68M draft model can be paired with various target models including Vicuna-7B, Qwen2-7B and Llama3-8B models for cross-vocabulary speculative decoding and provides up to 1.5-2x speedup on reasoning, coding and text generation tasks. Contributions (1) We propose cross-vocabulary speculative decoding via online n-gram cache that translates between drafter and target vocabularies, enabling speculative decoding across models with different tokenizers; (2) We introduce online knowledge distillation with hybrid alignment loss, which updates the drafter using accepted and corrected outputs