Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs

Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy, Sunwoo Lee · 2026

Read Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs by Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy, Sunwoo Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15491.txt.

Introduction

Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy, and Sunwoo Lee fix the harm from pruning whole layers of large language models. Cut layers leave the next live layer with inputs unlike those it trained on. The team works out a closed-form linear map from a small tuning set. It rebuilds the gap left by cut layers. Past no-train fixes stay inside tight map families. Theirs hits the free best of the alignment goal. Tests show gains in accuracy and perplexity on varied backbones and prune plans.

Catalog record

CREATORVincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy, Sunwoo Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90910
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15491

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs Vincent-Daniel Yun, Junhyuk Jo, Sai Praneeth Karimireddy, Sunwoo Lee Abstract Layer pruning removes entire Transformer decoder blocks from large language models, but introduces a mismatch between the hidden state received by the next surviving layer and the distribution it was trained to process, leading to significant performance degradation. We propose Ghosted Layers, a training-free recovery module that addresses this issue by solving a boundary activation alignment problem. Our method derives a closed-form optimal linear operator from a small calibration set to reconstruct the activation discrepancy introduced by the pruned layers. We show that this solution corresponds to the unconstrained optimum of the alignment objective, whereas existing methods are restricted to constrained solutions over limited operator subspaces. Experiments across multiple LLM backbones and pruning strategies demonstrate that our method consistently improves accuracy and perplexity over prior training-free baselines, while preserving the efficiency gains of layer pruning. 1 Introduction Large language models (LLMs) have shown strong capabilities across a wide range of natural language tasks ( 3 ; 33 ; 26 ) , but their size makes deployment costly. Various compression techniques have been explored to address this, including unstructured pruning ( 10 ; 32 ) and layer pruning ( 22 ; 14 ; 17 ; 31 ) . Among these, layer pruning stands out for its practicality, as it removes entire Transformer decoder blocks and yields a smaller model that runs on standard inference stacks without custom kernels or architectural changes. Yet pruned models often exhibit substantial accuracy degradation: pruning a block of layers eliminates the intermediate transformations between the surviving layers, causing a distribution shift at the pruning boundary that propagates through downstream layers. This motivates the need for a recovery mechanism to compensate for the missing layers. Recent work mitigates this through a variety of training-free recovery modules. ReplaceMe ( 30 ) approximates the pruned block’s computation with a linear transformation absorbed into the surviving weights, but does not directly address the mismatch at the pruning boundary. In contrast, LinearPatch ( 5 ) directly targets the boundary activation mismatch , the discrepancy between the hidden state expected by the next surviving layer and the one it actually receives, by inserting a linear operator at the pruning boundary. This operator is implemented as a Hadamard-rotated channel-wise scaling and is symmetric by construction, restricting it to a strict subspace of linear operators and preventing it from reaching the optimum of the alignment objective in the full operator space. To empirically verify this limitation, we quantify the boundary activation mismatch via the boundary activation error , defined as the mean absolute error (MAE) between the post-boundary hidden state of the original model and the input received by the next layer in the pruned model. Figure 1 compares representative post-pruning recovery methods on the resulting boundary activation error. Existing methods leave a substantial portion of the error uncorrected, suggesting that the boundary activation mismatch is not entirely resolved after recovery. Figure 1: Mean absolute error between the expected boundary activation and the activation received by downstream layers after pruning on LLaMA-3.1-8B. The pruned model is obtained using LLM-Streamline ( 4 ) with n=7 layers removed. We compare existing post-pruning recovery methods, including Prune&Comp ( 6 ) , ReplaceMe ( 30 ) , and LinearPatch ( 5 ) , against our Ghosted Layers. We address this limitation by deriving the closed-form unconstrained optimum of the boundary activation alignment objective over the full space of linear operators. In contrast, LinearPatch corresponds to a constrained solution restricted to a specific operator subspace. We further show that the optimal operator contains a substantial anti-symmetric component, which is structurally inaccessible to symmetric constructions such as LinearPatch . Building on this analysis, we propose Ghosted Layers , a training-free recovery module that inserts the closed-form optimal operator at the pruning boundary via a forward hook. It is compatible with any pruning criterion and model architecture, and consistently improves recovery across multiple LLM backbones and benchmarks. These results suggest that solving boundary activation alignment at its unconstrained optimum is sufficient to substantially recover layer-pruned LLMs without retraining. Contribution of this study can be summarized as follows: • We formulate post-pruning recovery as an activation alignment problem and derive its closed-form optimal solution from boundary activations. • We show that this formulation yields an unconstrained optimum that includes a substantial anti-symmetric component, which is inaccessible to symmetric constructions such as LinearPatch . • We propose Ghosted Layers , a training-free and plug-and-play recovery module compatible with any pruning criterion and model architecture, which consistently outperforms prior methods at matched inference cost. 0 0 footnotetext: Official code repository will be released soon. 2 Related works Structured pruning of LLMs. Structured pruning reduces model size by removing groups of parameters while preserving dense computation, enabling deployment without specialized kernels or sparse runtime support. Width pruning removes attention heads, MLP neurons, or hidden dimensions using importance scores derived from weights ( 20 ) , activations ( 1 ; 2 ) , or gradients ( 36 ) , but often introduces architectural irregularities and requires retraining or distillation to recover performance ( 25 ) . In contrast, depth pruning removes entire Transformer blocks while preserving the original architecture and requiring no specialized hardware support. Layer pruning. Various metrics have been proposed to identify redundant layers. ShortGPT ( 22 ) uses a Block Influence (BI) score based on cosine similarity between input and output activations for one-shot pruning. SLEB ( 31 ) iteratively removes layers that minimally increase perplexity on a calibration set. Shortened LLaMA ( 17 ) evaluates each layer via its perplexity impact under removal. LLM-Streamline ( 4 ) instead removes a contiguous block of layers with high boundary activation similarity. These methods primarily focus on selecting which layers to remove, leaving boundary mismatch to be handled separately. Post-pruning recovery. Prune&Comp ( 6 ) rescales the surviving boundary weights with per-channel scalars, capturing only channel-wise magnitude changes and no cross-channel interaction. ReplaceMe ( 30 ) instead approximates the pruned block’s computation with a linear map of the boundary MLP output, absorbed into the surviving MLP weights, so the repair acts on the block’s computation rather than on the boundary hidden state itself. LinearPatch ( 5 ) directly targets the boundary activation mismatch with a symmetric linear operator parameterized as a Hadamard-rotated diagonal scaling, confining the search to a strict subspace of \mathbb{R}^{C\times C} . Our Ghosted Layers shares the goal of reducing the boundary activation mismatch, but operates over the unconstrained space of linear maps and solves the resulting alignment problem in closed form, recovering structure that the channel-wise, block-level, and symmetric parameterizations above cannot express. 3 Method: Ghosted Layers Layer pruning removes transformer blocks to reduce model size, but the hidden state passed to the next surviving layer no longer matches the one it was trained to process, causing an activation mismatch at the pruning boundary that degrades downstream performance. We propose Ghosted Layers , a training-free recovery method that mitigates this mismatch via a closed-form linear operator inserted at each pruning boundary (Figure 2 ). Figure 2: Ghosted Layers as drop-in replacements for pruned transformer blocks. One or more consecutive transformer blocks are

Read the complete text