LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding

Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, Alexander Golubev · 2026

Read LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding by Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, Alexander Golubev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-23881.txt.

Introduction

Alexander Samarin and five co-authors present LK losses, training objectives that directly optimize the acceptance rate in speculative decoding. The 2026 paper notes that speculative decoding speeds large language models by letting a small draft model propose tokens that a large model verifies in parallel. Speed hinges on how often drafts are accepted.

Standard training minimizes KL divergence as a proxy, but small draft models often land where lower divergence does not mean higher acceptance. LK losses target acceptance itself, with no extra compute and easy integration into existing draft training. Tests across several draft and target models, from small to very large, and across general, coding, and math domains, show consistent acceptance gains over KL training. The method offers a direct, drop-in improvement for speculative decoding pipelines.

Catalog record

CREATORAlexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, Alexander Golubev
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-23881

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, Alexander Golubev Abstract Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to propose candidate tokens that are then verified in parallel by the target model. The speedup is significantly determined by the acceptance rate, yet standard training minimizes Kullback-Leibler (KL) divergence as a proxy objective. While KL divergence and acceptance rate share the same global optimum, small draft models, having limited capacity, typically converge to suboptimal solutions where minimizing KL does not guarantee maximizing acceptance rate. To address this issue, we propose LK losses , special training objectives that directly target acceptance rate. Comprehensive experiments across four draft architectures and six target models, ranging from 8B to 685B parameters, demonstrate consistent improvements in acceptance metrics across all configurations compared to the standard KL-based training. We evaluate our approach on general, coding and math domains and report gains of up to 8-10% in average acceptance length. LK losses are easy to implement, introduce no computational overhead and can be directly integrated into any existing speculator training framework, making them a compelling alternative to the existing draft training objectives. 1 Introduction Figure 1 : Acceptance length \tau vs maximum length K for EAGLE-3 draft models, trained using different objectives with Qwen3-235B-A22B-Instruct as a target model. The values were obtained on the MT-bench dataset with chain sampling and temperature = 1. Large language model (LLM) inference is fundamentally constrained by memory bandwidth rather than computational throughput. The autoregressive nature of token generation requires sequential memory accesses that underutilize modern accelerators, creating a critical bottleneck for deployment at scale. Speculative decoding ( 12 ; 4 ) addresses this challenge through a draft-then-verify paradigm: a lightweight draft model proposes multiple candidate tokens, which the target model then verifies in a single forward pass. This approach preserves the output distribution of the target model while achieving substantial speedups. The efficiency of this process is largely determined by the acceptance rate – an expected probability of a drafted token being accepted by the target model. A variety of draft model architectures implementing that idea have emerged, including parallel prediction heads ( 2 ; 16 ) , autoregressive draft heads with feature fusion ( 14 ; 15 ) , and multi-token prediction modules that can be repurposed for speculation after training ( 8 ) . These approaches commonly train draft models by minimizing Kullback-Leibler (KL) divergence between target and draft distributions, treating distributional alignment as a proxy for acceptance rate optimization. At the global optimum draft matches target perfectly, so this proxy is exact: KL divergence reaches zero and acceptance rate reaches one. However, draft models operate under severe capacity constraints, typically having 1-5% of the target model parameters, and inevitably converge to suboptimal solutions. At these suboptimal points, minimizing KL divergence provides no guarantee of maximizing acceptance rate. 12 noted that greater improvements might be obtained via custom training procedures, including direct optimization of the draft model for the acceptance rate, yet this direction has remained largely unexplored. In this work, we propose two variants of LK losses 1 1 1 The naming LK positions the loss as an alternative to KL divergence: standard approaches minimize KL as a proxy for acceptance rate, while LK objectives target acceptance directly. The notation also connects to the D_{LK}\equiv\text{TV} divergence in the original speculative decoding paper ( 12 ) . , training objectives that directly target acceptance rate. One of them is motivated by the maximum likelihood methods, directly optimizing the negative log-acceptance rate. The other variant is a hybrid approach, which gradually shifts the focus from KL towards direct acceptance optimization as training progresses, analogous to trust-region methods that balance a stable surrogate objective against the true optimization target. Both losses help improving average acceptance length, especially for longer draft sequences (see Figure 1 ). In summary, our contributions are as follows: • We propose two loss variants for direct acceptance rate optimization in speculative decoding. • We demonstrate consistent improvements in acceptance metrics across a number of target models and draft architectures, empirically confirming that LK losses are both model- and architecture-agnostic. • We release our training datasets and draft model weights to facilitate reproducibility and further research 2 2 2 Datasets: HuggingFace nebius/infinity-instruct-completions , Weights: HuggingFace nebius/lk-speculators . . 2 Related Work 2.1 Draft Model Architectures Early works on speculative decoding utilized smaller, standalone versions of the target model as draft models ( 12 ; 4 ) . Although straightforward to implement, such an approach usually suffers from performance bottlenecks within the draft model. Moreover, to achieve high acceptance rate it either needs a small pretrained language model (LM) from the same family or training from scratch on a sufficiently large corpus of data. To mitigate these overheads, subsequent works integrated the drafting mechanism directly into the target model. MEDUSA ( 2 ) attaches parallel decoding heads to the target model’s final layer to predict draft tokens independently from each other. This approach is computationally efficient but implicitly assumes conditional independence between the proposed tokens, which can degrade performance for distant draft positions. Other methods introduce autoregressive draft heads that also operate in the target model’s hidden state space. 18 propose a multi-stage MLP speculator that extends MEDUSA heads with the principles from recurrent networks. A more advanced EAGLE ( 14 ; 15 ) family employs a shallow transformer model with a causal mask to better capture long-term dependencies between draft tokens and their context. EAGLE-3 also enriches the input feature space of the speculator heads by fusing hidden states from various intermediate layers of the target model. A similar autoregressive design was recently adopted by DeepSeek-V3 with its Multi-Token Prediction (MTP) module ( 8 ) . It predicts multiple draft tokens during training, essentially serving as a native “draft head” that requires no separate post-training. In addition to architectural advances, some works address the problem of computational overhead in the LM heads of the draft models. FR-Spec ( 21 ) noticed that large token vocabularies of contemporary LLMs often make the latency of LM heads dominate in the total latency of the speculator heads, even in such powerful approaches as EAGLE. FR-Spec addresses this issue by truncating the draft vocabulary to a small subset of high-frequency tokens learned on training data. 2.2 Training Methodologies for Draft Models Knowledge Distillation. The problem of training speculators has been predominantly framed as the Knowledge Distillation (KD). Within this paradigm, draft models are students that approximate the teacher’s (i.e. target) output distribution as closely as possible. The standard training objective has therefore been KL divergence, or equivalently cross-entropy (CE), between the target and draft distributions. Some works experimented with combined objectives. MEDUSA suggests mixing the KL loss with the LM objective for the target model to address the discrepancy between its distribution and static training data. EAGLE ( 14 ) extends KL with regression loss on hidden states of the last layer in an attempt to better match training and inference settings. Targeting Acceptance Rate. Some studies went beyond the standard KD framework in terms of both training data and objectives. DistillSpec ( 23

Read the complete text