TIDE : Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference
Jaber Jaber, Osama Jaber · 2026
Read TIDE : Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference by Jaber Jaber, Osama Jaber (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21365.txt.
Introduction
Jaber and Jaber present TIDE, a system that lets tokens exit early during language model inference. Models normally run every token through every layer, whatever its difficulty. TIDE attaches tiny learned routers at checkpoint layers. At run time, each token exits at the earliest layer where its hidden state has converged. No retraining is needed. On an A100 with an 8B model, nearly all decoding tokens exit early, throughput rises by several percent, and calibration takes under three minutes.
Catalog record
| CREATOR | Jaber Jaber, Osama Jaber |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23499 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21365
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TIDE : Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference Jaber Jaber, Osama Jaber Abstract Large language models run every token through every layer, regardless of difficulty. A function word like “the” receives the same 32-layer treatment as a reasoning step in a math derivation. We present Tide , a post-training system that attaches tiny learned routers at periodic checkpoint layers and, at inference time, selects the earliest layer whose hidden state has converged for each token. Tide requires no model retraining, works with any HuggingFace causal LM, auto-detects GPU architecture, and supports float32, float16, and bfloat16 through fused CUDA kernels. On an NVIDIA A100 with DeepSeek R1 Distill 8B, Tide achieves 100% prefill exit rate (5% of tokens exit at layer 11, the remaining at layer 31), reduces prefill latency by 7.2%, and increases single-batch throughput by 6.6%. During autoregressive decoding, 98–99% of tokens exit early while the model correctly solves a multi-step math problem with 95 unique output tokens. On Qwen3 8B (36 layers), throughput improves by 8.1% at batch size 8. Calibration on 2,000 WikiText samples takes under 3 minutes and produces a ∼ {\sim} 4 MB router checkpoint. The system comprises 1,308 lines of Python and 1,081 lines of CUDA/C++ with 74 passing tests. Code and package: https://github.com/RightNow-AI/TIDE 1 Introduction Transformer-based language models allocate identical compute to every token at every position. A 32-layer model performs 32 matrix multiplications per token whether that token carries critical semantic content or is a stopword repeated thousands of times in the training corpus. At scale, this uniform allocation is wasteful: prior work on representation similarity ( Schuster et al. 2022 ) shows that for a large fraction of tokens, intermediate hidden states become nearly identical to the final hidden state well before the last layer. The compute spent on those remaining layers produces negligible change in the output distribution. The cost is concrete. Serving a 70B-parameter model on 8 GPUs costs $3–5 per hour. Prefill latency for a 4,096-token prompt on an A100 takes 100–200 ms, and autoregressive decode dominates end-to-end latency for long outputs. Reducing depth per token by even a few layers translates directly to lower latency, higher throughput, and reduced energy consumption. Early exit methods exist but face three gaps. First, encoder-only approaches like DeeBERT ( Xin et al. 2020 ) and FastBERT ( Liu et al. 2020 ) target BERT classification and do not generalize to autoregressive generation with KV caches. Second, methods that require early-exit pretraining ( Elhoushi et al. 2024 ; Chen et al. 2023b ) demand access to the training pipeline and hundreds of GPU-hours, making them impractical for users of pretrained checkpoints from model hubs. Third, confidence-based heuristics ( Zhou et al. 2020 ; Schuster et al. 2022 ) use the model’s own softmax entropy as an exit signal, which is unreliable for generation where entropy is naturally high. Tide takes a different approach. We train lightweight binary classifiers (routers) on top of frozen model hidden states, using cosine similarity between each checkpoint layer and the final layer as the convergence signal. Each router is a two-layer MLP with a bottleneck dimension of 128, totaling d\times 128+128 parameters per checkpoint (524,416 parameters for d{=}4{,}096 ). Training takes 100 epochs of binary cross-entropy on 2,000 WikiText samples, completing in under 3 minutes on a single GPU. The resulting router checkpoint is {\sim} 4 MB. The key insight is that convergence is a property of the token, not the model. The same model will show different convergence patterns for different inputs. A learned router captures this token-level signal more reliably than a global heuristic like softmax entropy or patience-based counting. Our contributions: 1. A post-training early exit system for autoregressive LLMs that requires no model modification and works with any HuggingFace causal language model (Section 3 ). 2. A universal model adapter that auto-probes transformer structure across 17 attribute paths, covering LLaMA, GPT-2, GPT-NeoX, Phi, Falcon, OPT, and any other architecture (Section 3.2 ). 3. Fused CUDA kernels for RMSNorm + router evaluation in a single launch, with native fp16/bf16 support and 8 template specializations for common hidden dimensions (Section 3.4 ). 4. A post-hoc exit strategy for autoregressive generation that preserves KV cache integrity and is compatible with all versions of the transformers library (Section 3.3 ). 5. Empirical evaluation on DeepSeek R1 Distill 8B and Qwen3 8B showing 100% prefill exit rate, 99% decode exit rate, and up to 8.1% throughput improvement on an A100 (Section 5 ). 6. An open-source release with 74 tests, PyPI packaging ( pip install tide-inference ), and GPU auto-detection from V100 through Blackwell (Section 5.6 ). 2 Related Work Early exit in encoder models. DeeBERT ( Xin et al. 2020 ) adds classifiers at each BERT layer and exits when confidence exceeds a threshold. FastBERT ( Liu et al. 2020 ) uses self-distillation to train branch classifiers. PABEE ( Zhou et al. 2020 ) introduces patience-based exit: a token exits after k consecutive layers agree on the prediction. BranchyNet ( Teerapittayanon et al. 2016 ) pioneered exit branches in CNNs and early DNNs. These methods target discriminative tasks and do not handle autoregressive generation or KV caches. Early exit in decoder models. CALM ( Schuster et al. 2022 ) applied confidence-based early exit to encoder-decoder T5 models and showed 2–3 \times speedup on translation. LayerSkip ( Elhoushi et al. 2024 ) trains decoder-only models with early-exit loss from scratch using a layer dropout schedule. EE-LLM ( Chen et al. 2023b ) integrates early exit into the pretraining pipeline of large language models. Shan et al. ( Shan et al. 2024 ) study early exit as a natural capability of pretrained LLMs without additional training. Tide differs from all of these: it requires no retraining, uses learned routers rather than confidence heuristics, and supports any pretrained checkpoint. Adaptive computation. Graves ( Graves 2016 ) introduced Adaptive Computation Time (ACT), which learns a halting probability per step. Universal Transformers ( Dehghani et al. 2019 ) apply ACT to shared-weight transformer layers. Mixture-of-Depths ( Raposo et al. 2024 ) routes tokens to skip entire layers during training. ADEPT ( Yoo et al. 2026 ) uses a draft model to decide token depth. Han et al. ( Han et al. 2022 ) survey dynamic neural networks broadly. Tide applies depth-wise sparsity post-training, without modifying the model architecture or training procedure. Speculative decoding. Speculative decoding ( Leviathan et al. 2023 ; Chen et al. 2023a ) uses a small draft model to generate candidate tokens, then verifies them in parallel with the full model. EAGLE ( Li et al. 2024 ) autoregressively generates draft tokens from features. SpecEE ( Xu et al. 2025 ) combines speculative decoding with early exit. These methods add a separate draft model. Tide achieves early exit within the target model itself, with no auxiliary model overhead. KV cache optimization. Layer-Condensed KV Cache ( Wu and Tu 2024 ) shares KV pairs across layers to reduce memory. SkipDecode ( Del Corro et al. 2023 ) skips lower layers during decode, accepting the KV cache discontinuity. Tide runs all layers in post-hoc mode (preserving cache integrity) while selecting the exit layer’s hidden state for logit