LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models

Prateek Kumar Sikdar · 2026

Read LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models by Prateek Kumar Sikdar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01838.txt.

Introduction

Sikdar cuts wasted compute in agent-style language models. Agent work mixes two kinds of steps. Tool calls are short, fixed, and easy to predict. Planning steps are long, complex, and hard to predict. Yet inference stacks spend the same compute on both.

His LayerRoute tool learns when to skip layers. Each block of a small Qwen model gains a tiny router. LoRA adapters tune the attention parts per block. The base model weights stay frozen throughout. One training pass uses mixed agent data sets. Tool calls end up skipping more compute than plans. The skip gap is 12.91 percent, Sikdar reports. Training took just 6.4 minutes on one A100 chip.

Catalog record

CREATORPrateek Kumar Sikdar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01838

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models Prateek Kumar Sikdar Abstract Agentic language model systems alternate between two structurally distinct step types: structured tool calls (short, deterministic, low perplexity) and open-ended planning/reasoning steps (long, complex, high perplexity). Despite this heterogeneity, current inference systems apply identical compute to every step. We introduce LayerRoute , a lightweight adapter that learns to selectively skip transformer blocks on a per-input basis. LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with: (1) a per-layer router ( ∼ \sim 897 parameters, Linear(896,1)) that outputs a hard binary gate via the straight-through estimator, and (2) LoRA adapters (rank 8, ∼ \sim 1.08M parameters) on the Q/K/V/O attention projections. The backbone weights remain frozen. A single end-to-end training pass on agentic data (Hermes, Glaive, GSM8K, Turing) with a gate regularisation term forces the system to discover which blocks are skippable per input type. After 3,000 steps (6.4 minutes on an A100 40GB), LayerRoute achieves a 12.91% skip differential : tool calls skip 15.25% of FLOPs while planning steps skip only 2.34%, using only 1.10M trainable parameters (0.22% of the 494M backbone). Quality improves over the base model due to LoRA adaptation, with perplexity delta of − 1.29 -1.29 on tool calls and − 1.30 -1.30 on planning. 1 Introduction Modern agentic AI systems orchestrate large language models (LLMs) across multi-step workflows: querying APIs, executing tool calls, reasoning over retrieved documents, and generating structured outputs Yao et al. 2023 ; Schick et al. 2023 . A critical but underexplored inefficiency arises from the heterogeneous nature of these steps. A tool call—such as search_database(table=’products’, filter=’id=100023’) —is a short, structured sequence where the next token is highly predictable. A planning step—such as “ Develop a prioritization strategy for declining user retention across product segments ”—requires complex multi-step reasoning over a broad output space. Current inference systems treat both step types identically, routing every input through all transformer layers at full compute cost. This is wasteful: for tool calls, the model’s high confidence early in the residual stream suggests that deep layers contribute marginally to the final prediction. We address this with LayerRoute , a parameter-efficient adapter that learns input-conditioned layer skipping. Our contributions are: 1. Per-layer hard-gated skip connections with straight-through estimation (STE), eliminating train/inference mismatch. 2. Joint LoRA fine-tuning of attention projections alongside router learning, enabling the router to discover task-relevant layer importance via the language modelling objective directly. 3. Gate regularisation to prevent the router from collapsing to a trivial solution where all gates remain open. 4. Biased initialisation of middle-layer routers to break symmetry and enable gradient-based differentiation from the first training step. 5. Empirical validation on Qwen2.5-0.5B-Instruct showing a 12.91% skip differential between tool-call and planning steps, trained in under 7 minutes. 2 Related Work Early exit and layer skipping. DeeBERT Xin et al. 2020 and PABEE Zhou et al. 2020 enable early exit from transformer encoders based on intermediate confidence. SkipBERT Tang et al. 2023 learns to skip entire layers via input-dependent routing. Mixture of Depths (MoD) Raposo et al. 2024 applies per-token routing to control compute allocation across transformer depth. Our approach shares the skip-connection mechanism with MoD but differs in training strategy: we use LoRA joint fine-tuning rather than training from scratch, and target agentic step types rather than generic language modelling. Parameter-efficient fine-tuning. LoRA Hu et al. 2022 decomposes weight updates into low-rank matrices, enabling efficient adaptation with minimal parameters. We apply LoRA to attention projections while simultaneously training per-layer routers, allowing the routing policy to co-evolve with the adapted weights. Efficient inference for LLMs. Speculative decoding Leviathan et al. 2023 , PagedAttention Kwon et al. 2023 , and FlashAttention Dao et al. 2022 improve throughput without modifying model architecture. LayerRoute is complementary—it reduces active layer count rather than optimising attention kernels or memory. Agentic LLM systems. AgenticQwen Alibaba 2026 trains Qwen models for agentic tasks via GRPO and data flywheels, demonstrating that agentic fine-tuning substantially improves task performance. LayerRoute is complementary: we target inference efficiency rather than task capability. 3 Method 3.1 Architecture LayerRoute augments a frozen pretrained transformer (Qwen2.5-0.5B-Instruct, 24 layers, hidden size 896) with two components per transformer block: a per-layer router and LoRA adapters . Per-layer router. Each router r_{i} ( i=0,\ldots,23 ) is a lightweight linear layer: s_{i}=\mathbf{w}_{i}^{\top}\bar{\mathbf{h}}_{i}+b_{i},\quad\sigma_{i}=\sigma(s_{i}),\quad g_{i}\in\{0,1\} (1) where \bar{\mathbf{h}}_{i}\in\mathbb{R}^{d} is the mean-pooled hidden state entering block i , \mathbf{w}_{i}\in\mathbb{R}^{d} , and \sigma(\cdot) is the sigmoid function. The hard gate is: g_{i}=\mathbf{1}[\sigma_{i}>0.5] (2) Each router has d+1=897 parameters; the collection of 24 routers totals 21,528 parameters. Gated skip connections. The forward pass through block i is: \mathbf{h}_{i+1}=g_{i}\cdot\text{Block}_{i}(\mathbf{h}_{i})+(1-g_{i})\cdot\mathbf{h}_{i} (3) When g_{i}=1 the block runs normally (with LoRA adapters active). When g_{i}=0 the hidden state passes through unchanged, incurring zero block compute. LoRA adapters. For each attention projection \mathbf{W}\in\{W_{Q},W_{K},W_{V},W_{O}\} in each block, we add low-rank adapters: \mathbf{W}^{\prime}=\mathbf{W}+\frac{\alpha}{r}\mathbf{B}\mathbf{A},\quad\mathbf{A}\in\mathbb{R}^{r\times d_{in}},\;\mathbf{B}\in\mathbb{R}^{d_{out}\times r} (4) with rank r=8 , \alpha=16 . \mathbf{B} initialised to zero so LoRA starts as identity. Total LoRA parameters: 4\times 24\times 2\times 896\times 8=1{,}081{,}344 . 3.2 Straight-Through Estimator (STE) The hard thresholding g_{i}=\mathbf{1}[\sigma_{i}>0.5] is non-differentiable. We apply the straight-through estimator Bengio et al. 2013 : \hat{g}_{i}=\underbrace{\mathbf{1}[\sigma_{i}>0.5]}_{\text{forward}}-\underbrace{\sigma_{i}}_{\text{stop-grad}}+\underbrace{\sigma_{i}}_{\text{backward}} (5) The forward pass uses the hard gate; gradients flow through \sigma_{i} as if it were continuous. This eliminates train/inference mismatch: both training and inference use identical hard \{0,1\} decisions. 3.3 Training Objective All trainable parameters—LoRA matrices \{\mathbf{A}_{i},\mathbf{B}_{i}\} and router weights \{\mathbf{w}_{i},b_{i}\} —are optimised jointly via: \mathcal{L}=\mathcal{L}_{\text{LM}}+\lambda\cdot\frac{1}{L}\sum_{i=0}^{L-1}\sigma(s_{i}) (6) where \mathcal{L}_{\text{LM}} is the standard autoregressive cross-entropy loss and the second term is gate regularisation with weight \lambda=1.0 . Gate regularisation. Without the regularisation term, the router has no pressure to close gates—LoRA can adapt weights to maintain quality even with all layers active, leaving the gate loss gradient near zero. The regularisation penalises uniformly high soft gate values, forcing the router to identify genuinely skippable blocks. Crucially, the penalty acts on \sigma(s_{i}) (the soft sigmoid, in the backward path) while the forward pass remains hard \{0,1\} . There is no mismatch. 3.4 Biased Initialisation Uniform initialisation places all gates near \sigma(0)=0.5 , creating a chicken-and-egg problem: gates will not differentiate until they begin skipping, but they will not skip until they differentiate. We break this symmetry by initialising the router biases differently by layer position: b_{i}=\begin{cases}+1.0&i\in\{0\text{--}7,\;17\text{--}23\}\quad[\sigma(b_{i})\approx 0.73]\\ -1.0&i\in\{8\text{--}16\}\quad[\sigma(b_{i})\approx 0.27]\end{cases} (7) Middle layers start below the 0.5 threshold and skip from step 1, immediately exposing their contribution (or lack thereof) to the LM loss. Early and late layers start open, consistent with the known importance of early embedding layers and late output layers in transformer architectures Geva et al. 2021 . 3.5 Architecture Summary Table 1 summarises the trainable parameter count. Figure 2 illustrates the full architecture. Table 1: LayerRoute trainable parameter breakdown. Component Parameters % of backbone LoRA adapters (r=8, Q/K/V/O, 24 layers) 1,081,344 0.219% Per-layer routers (24 \times Linear(896,1)) 21,528 0.004% Total trainable 1,102,872 0.223% Frozen backbone (Qwen2.5-0.5B) 494,032,768 — 4 Experiments 4.1 Setup Base model. Qwen2.5-0.5B-Instruct: 24 transformer blocks, hidden size 896, 14 query heads / 2 KV heads (GQA), SwiGLU FFN (intermediate size 4,864), vocabulary 151,936, RoPE positional encoding. Training data. We construct a mixed agentic dataset: • Tool-call : NousResearch Hermes Function Calling v1 (1,893 samples), Glaive Function Calling v2 (5,000 samples) •

Read the complete text