EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

2026

Read EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27550.txt.

Introduction

Chen speeds up language model inference, in a machine learning paper published in 2026, by letting speculation depth follow the local entropy of the text being generated. Multi-token prediction heads draft several future tokens at once for parallel verification, but existing systems fix one tree-shaped drafting pattern for the whole sequence. Natural language is uneven: predictable stretches support deep drafting, while uncertain stretches need caution. EntMTP, a training-free scheduler, switches among a set of task-specific Pareto-optimal drafting trees using a running estimate of generation entropy. No retraining of the underlying model is required.

Across code, chat, mathematics, and literary benchmarks, the scheduler consistently beats a leading tree-based baseline by a factor between about one point one and one point one five in the reported speedups. Against an earlier drafting method the peak speedup reaches about one point three six times in the same evaluation.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27550

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction Abstract Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding. Existing foundation and open source models that use MTP heads commit to a static tree-based attention topology throughout the entire generation sequence, meaning the speculation depth, and thus the compute required during verification, stays constant regardless of the context. This is fundamentally misaligned with the entropy patterns of natural language where low-entropy regions often support reliable multi-step drafting, while high-entropy regions require more conservative speculation. To address this, we propose Entropy-guided Multi-Token Prediction (EntMTP), a training-free scheduler that toggles between tree-based attention topologies from a set of task-specific pareto-optimal trees conditioned on a running estimate of local generation entropy. By matching speculation depth to context predictability, EntMTP maximizes expected accepted-token throughput across the full distribution of generated text without sacrificing generation quality. When evaluated across Humaneval, ShareGPT, GSM8k, and Litbench benchmarks, EntMTP consistently achieves a 1.09-1.15x speedup against Hydra and peak speedup of ∼ \sim 1.36x against Medusa baselines respectively. Carrie Chen Cornell University [email protected] 1 Introduction Large language models (LLMs) have achieved incredible performance across a broad range of tasks, however the sequential nature of autoregressive decoding remains a bottleneck to their latency: each generated token requires a full forward pass through all model parameters, making inference memory-bandwidth bound and expensive at scale. Speculative decoding ( 8 ; 3 ) addresses this by drafting k candidate tokens with a cheap proposal model and verifying them in a single parallel pass of the target model, preserving the target distribution exactly. Multi-token prediction (MTP) heads ( 2 ; 1 ) implement speculative decoding without a separate draft model by attaching lightweight modules to the target model’s final hidden state and generating a tree of candidates for sparse verification. For MTP-based decoding, performance is largely determined by the realized accept depth at each step. Prior work ( 2 ; 1 ) uses this metric to select a draft-tree topology offline on a subsample of Alpaca ( 7 ) ; the selected topology is then reused throughout inference. We show in Appendix A that this global selection is too coarse. Draft-token acceptance varies with position, local context, and overlap with the training distribution. More importantly, both acceptance behavior and the throughput frontier over tree topologies are task-specific : a topology that is Pareto-optimal for one benchmark can be highly suboptimal for another. We propose EntMTP, a lightweight runtime tree-selection layer that exploits this task dependence rather than treating it as noise. EntMTP first builds a throughput-Pareto frontier of Hydra tree topologies offline, using greedy expansion and self-rollout acceptance estimates. It retains only frontier trees that improve the cost–throughput tradeoff, discarding dominated topologies. At inference time, a precompiled TopologyBank stores the attention masks, position offsets, and gather indices for each frontier tree. Switching topologies therefore reduces to an O(1) pointer swap, with no mask reconstruction or kernel rebuild. At each decode step, the scheduler reads features off-the-shelf from the model, such as the EAGLE-2 path value, or the base top-1 probability, and selects a tree using one of three policies: EntMTP- l , a threshold ladder, EntMTP ∗ , a static-task-optimal tree rule, or EntMTP τ , a hysteretic binary- τ rule with separate on/off thresholds to prevent per-step jitter. In the batch\_size=1 setting, EntMTP τ dominates both the Hydra and Medusa baselines as well as EntMTP ∗ across all three evaluation tasks: grade school math, code completion, and conversational llm. On GSM8K it achieves 112.0 tok/s vs. 102.4 for Hydra ( +9.4\% ), with a speedup of 3.13\times over autoregressive decoding. On HumanEval it achieves 124.7 tok/s ( +14.0\% over Hydra, 3.26\times over AR). The largest absolute gain is on ShareGPT: 117.5 tok/s vs. 109.0 ( +7.8\% , 3.47\times AR). Response perplexity stays within 0.022 of Hydra across all settings, confirming that gains stem from scheduling, not from changes to the verification rule. 2 Related Work 2.1 Speculative Decoding Speculative decoding ( 8 ; 3 ) accelerates autoregressive generation losslessly by alternating a low-cost draft stage with a parallel verification stage on the target model. Given a prefix T_{1:j} , a draft distribution q produces \hat{T}_{j+1:j+k} together with its per-token probabilities \hat{p}_{j+i} , and the target distribution p then evaluates the entire candidate in a single forward pass. Tokens are accepted left-to-right with probability \min(1,\,p_{j+i}(\hat{t}_{j+i})/\hat{p}_{j+i}(\hat{t}_{j+i})) ; on the first rejection, a replacement is drawn from the residual \mathrm{norm}(\max(0,\,p_{j+i}-\hat{p}_{j+i})) and the suffix is discarded. This ensures that the marginal distribution of accepted tokens matches the target’s, so the realized speedup depends entirely on how well q mimics p and how parallelizable verification is. 2.2 EAGLE and EAGLE-2 EAGLE ( 6 ) replaces token-level drafting with feature-level autoregression. Given the target model’s hidden-state sequence F_{1:i} and token sequence T_{1:i} , a lightweight drafter predicts the next hidden state, \hat{F}_{i+1}=D_{\theta}(T_{2:i+1},F_{1:i}) which is then decoded by the target model’s frozen LM head: p_{\theta}(t_{i+1}\mid t_{\leq i})=\operatorname{softmax}(W_{\mathrm{LM}}\hat{F}_{i+1}) The drafter is trained to match the target model’s next hidden state using a feature-regression loss, \mathcal{L}_{\mathrm{reg}}=\operatorname{SmoothL1}\left(F_{i+1},D_{\theta}(T_{2:i+1},F_{1:i})\right) By drafting in hidden-state space rather than directly predicting independent future tokens, EAGLE obtains a stronger drafter at comparable parameter cost. EAGLE-2 ( 5 ) keeps this feature-level drafter but makes the draft tree dynamic. Instead of committing to a fixed topology, it scores each candidate path using the drafter’s own confidence. For a node t_{i} in the draft tree, its path value is defined as V_{i}=\prod_{t_{j}\in\operatorname{Path}(\mathrm{root},t_{i})}p_{j}\approx\prod_{t_{j}\in\operatorname{Path}(\mathrm{root},t_{i})}c_{j} where p_{j} is the true, unobserved acceptance probability at node t_{j} , and c_{j} is the drafter’s local confidence. At inference time, EAGLE-2 expands the tree top-down by repeatedly selecting the highest-value frontier nodes: t^{\star}=\arg\max_{t_{i}\in\mathcal{F}}V_{i} where \mathcal{F} denotes the current frontier. Since the scores c_{j} are already produced by the drafter, this dynamic tree construction requires no additional trained components and preserves lossless target-model verification. 2.3 Independent Multi-Head Prediction Medusa ( 2 ) takes a more direct approach to speculation by attaching K lightweight prediction heads to the target model’s final hidden state. Each head h_{i} predicts the token at offset i independently of the other drafted tokens, factorizing the draft distribution as \prod_{i}p_{h_{i}}(t_{j+i}\mid h_{j}) The heads emit top- k_{i} candidates, whose Cartesian product is arranged into a draft tree with precomputed attention masks, position offsets, and gather indices. Verification then reduces to a single sparse forward pass through the target model. 2.4 Sequential Multi-Head Prediction Hydra ( 1 ) keeps Medusa’s head-on-hidden-state design but restores sequential dependence across draft heads. Instead of predicting each future token independently, head h_{i} conditions on the embeddings of tokens already drafted by earlier heads, modeling p(t_{j+i}\mid t_{\leq j},\hat{t}_{j+1:j+i-1}) instead of the marginal p(t_{j+i}\mid t_{\leq j}) Each head is implemented as a small causal transformer block over the base hidden state and previously drafted token embeddings. This preserves an O(K) head cost while allowing the draft chain to condition on its own prefix. However, Hydra inherits Medusa’s static-tree limitation: a single topology is selected offline and reused for every decoding step. 3 Optimizing Task-Specific Greedy Draft Trees Hydra’s acceptance behavior is workload-dependent in ways the verifier and base model cannot absorb. On ShareGPT , recent acceptance history

Read the complete text