Depth Exploration for LLM Decoding

Weisi Yang, Zipeng Sun, Stephen Xia · 2026

Read Depth Exploration for LLM Decoding by Weisi Yang, Zipeng Sun, Stephen Xia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29223.txt.

Introduction

Yang, Sun and Xia propose Depth Exploration Decoding, a way to generate text from language models faster without changing the output. Standard decoding runs every token through the full layer stack, although many tokens become predictable at intermediate depths. Earlier depth-adaptive methods pick one early exit depth and verify it, but a late exit wastes computation and an early one triggers fallback.

Their algorithm explores multiple candidate depths in parallel instead of selecting one. At each commit position it checks candidates against the final-depth reference and commits exactly the final-depth token, then collapses the exploration lattice, keeping only reusable branch states. This cycle preserves exact equivalence to standard decoding while cutting the cost per token. Across early-exit-trained and standard models, it beat depth-selection baselines and matched speculative and distributed decoding methods in throughput. It also improved as explored depths became finer.

Catalog record

CREATORWeisi Yang, Zipeng Sun, Stephen Xia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Depth Exploration for LLM Decoding Weisi Yang, Zipeng Sun, Stephen Xia Abstract Autoregressive LLM decoding evaluates every generated token through the full layer stack, even though many tokens become predictable at intermediate depths. Existing lossless depth-adaptive methods exploit this redundancy by choosing a single non-final exit depth and verifying its prediction with the final-depth model. However, our measurements show that this selection-based strategy leaves substantial headroom: choosing an exit too late wastes computation, while choosing one too early triggers fallback and discards dependent drafts. We propose Depth Exploration Decoding (DEX), a lossless decoding algorithm that replaces single-depth selection with parallel exploration over multiple candidate depths. At each commit position, DEX validates candidates against the final-depth reference, commits exactly the final-depth token, and collapses the exploration lattice to retain only reusable branch states. This expand–commit–collapse procedure preserves equivalence to standard autoregressive decoding while reducing the cost of committing each token. Across early-exit-trained and standard LLMs, DEX outperforms representative depth-selection baselines and achieves competitive end-to-end throughput against speculative and distributed decoding methods. Moreover, DEX improves as the explored depths become finer, showing that parallel depth exploration provides a scalable way to exploit the underused depth axis of LLM decoding. 1 Introduction The large parameter size of modern Large Language Models (LLMs) gives them strong capabilities across many domains, but also makes decoding expensive. A major source of this latency is autoregressive (AR) decoding: LLMs generate responses token by token, and each generated token must pass through the full layer stack before the next token can be produced. Recent studies [ 20 , 7 ] identify redundancy along the depth dimension, suggesting that not all tokens require the full model depth to become predictable. Leveraging this observation, prior work [ 8 , 25 , 5 ] has proposed depth-adaptive decoding methods that allow tokens to exit from intermediate layers, reducing the amount of layer computation and improving decoding latency. More recently, the draft-and-verify paradigm in speculative decoding [ 15 ] has been adopted by lossless depth-adaptive methods [ 9 , 28 , 33 ] . These methods draft candidates from non-final depths and verify them with the final-depth model, preserving equivalence with standard AR decoding while reducing layer computation. However, existing methods share a selection bottleneck that limits their potential speedup: they choose only a single exit depth for each token, either through a fixed or learned policy. Choosing too late wastes layers even when the token was already predictable earlier, while choosing too early can trigger final-depth replacement and discard descendant drafts. Our empirical studies confirm this gap between existing methods and an oracle that leverages the earliest intermediate depth whose prediction matches the full-depth output (Figure 3 ). This raises the key question: can we exploit depth redundancy by reducing the risk of a wrong exit-depth decision? We propose depth exploration ( DEX ), whose key idea is to replace single-depth prediction with multi-depth coverage. Instead of betting on one exit depth, DEX explores several candidate depths in parallel and uses the full-depth output as the reference to identify the earliest explored branch that matches it. A missed shallow candidate therefore does not immediately force a full-depth fallback: a deeper explored branch may still match and be reused. As a result, the error of depth adaptation is controlled by the spacing between explored depths rather than by the accuracy of a single exit-depth predictor. As the exploration set becomes finer, the first reusable branch moves closer to the earliest depth at which the token would have matched the full-depth output. The main challenge is not merely exposing more exits, but turning parallel depth coverage into an exact decoding procedure. Exploration creates multiple possible futures, yet AR decoding has only one official prefix. After each commit, the decoder must identify which branch computations remain valid under the new prefix and which have become stale. DEX addresses this with an expand–commit–collapse procedure: it expands candidate branches across depth stages, commits only the final-depth-validated token, and collapses the exploration lattice onto the branch consistent with that commit (Section 3.2 ). This allows DEX to reuse parallel depth exploration when it agrees with the final model, while preserving lossless equivalence to standard AR decoding. To conclude, our main contributions are: • We define Earliest Available Depth (EAD) to quantify depth-side token readiness. Building on EAD, we formulate depth selection and depth exploration, and identify the selection bottleneck in existing lossless depth-adaptive decoding methods. • We present DEX , a lossless decoding algorithm that instantiates depth exploration through parallel branch expansion, final-depth validation and commit, and collapse for clearing tokens from rejected branches. • We show that DEX outperforms prior depth-based baselines and achieves competitive end-to-end throughput against representative speculative and distributed decoding baselines, with stronger scaling as more depth explorers are available. Figure 1: Comparison of different depth exploitation. (Left) Autoregressive decoding always performs a full-depth forward pass. (Middle) Depth selection relies on a single exit-depth decision, risking either full-depth fallback when exiting too early or wasted computation when exiting too late. (Right) Parallel depth exploration over candidate depths resolves branches with the final-depth reference and preserves only reusable branches. 2 Preliminaries and problem formulation This section gives the compact formulation used by the method and experiments. Appendix A provides additional EAD measurements, illustrative examples, and full derivations. 2.1 Autoregressive decoding and token readiness. Consider a decoder-only language model with L layers that generates T new tokens. Standard autoregressive (AR) decoding will have LT layer computations. However, for many tokens, an intermediate layer may already produce a candidate that agrees with the final-depth decoding outcome. We use this agreement to define when a token becomes ready along the depth axis. Definition 1 (Acceptance and stable earliest available depth) . Let [L]=\{1,\ldots,L\} and [T]=\{1,\ldots,T\} . For token position t\in[T] and depth \ell\in[L] , let \Omega_{\mathcal{R}}(t,\ell)\in\{0,1\} denote whether the candidate token produced at depth \ell is accepted by the final-depth decoding rule \mathcal{R} . We set \Omega_{\mathcal{R}}(t,L)=1 by construction. The earliest available depth is \mathrm{EAD}_{\mathcal{R}}(t)=\min\{\ell\in[L]:\Omega_{\mathcal{R}}(t,\ell)=1\}. (1) Because intermediate predictions can occasionally match and then diverge at later layers, we use a stable version as the default quantity. Define stable acceptance by \Omega_{\mathcal{R}}^{\mathrm{stab}}(t,\ell)=1\quad\Longleftrightarrow\quad\Omega_{\mathcal{R}}(t,n)=1,\;\forall n\in\{\ell,\ell+1,\ldots,L\}. (2) The stable earliest available depth is \mathrm{EAD}^{\mathrm{stab}}_{\mathcal{R}}(t)=\min\{\ell\in[L]:\Omega_{\mathcal{R}}^{\mathrm{stab}}(t,\ell)=1\}. (3) Unless otherwise stated, EAD refers to this stable EAD and we omit the subscript \mathcal{R} when the decoding rule is fixed. 2.2 Depth selection and its bottleneck. Existing lossless depth-adaptive methods typically instantiate depth adaptation as a selection problem: they choose one non-final depth, draft a token from that depth, and rely on final-depth verification to preserve equivalence with AR decoding. Definition 2 (Depth selection) . A depth-selection policy assigns a selected non-final exit depth s_{t}\in[L-1] to each drafted token position t . Under the stable-readiness convention used in this analysis, the selected exit can exploit token readiness only when \Omega^{\mathrm{stab}}(t,s_{t})=1 , equivalently \mathrm{EAD}(t)\leq s_{t} . Let p_{t}^{\mathrm{acc}} denote the probability that the selected exit at s_{t} is accepted under this workload model. If this exit fails, final-depth verification must replace the token, and any drafted descendants that depend on the rejected token must be discarded. Let v_{t}\geq 1 be the number of discarded selected-exit tokens from position t onward, including token t . The

Read the complete text