TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination

Omar Naim, Krish Sharma, Nicholas Asher · 2025

Read TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination by Omar Naim, Krish Sharma, Nicholas Asher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22767.txt.

Introduction

Not all layers in a large model help on every task. Omar Naim, Krish Sharma, Niyar R Barman, and Nicholas Asher use that fact in 2025. Their method is called TALE. It removes task harmful layers at inference time. No retraining is needed.

TALE stands for Task-Aware Layer Elimination. Across nine tasks, it matched or beat baselines. It was tested on five model families. Both zero shot and few shot settings were used. It also cut compute costs at the same time. Fine tuning plus TALE gave further gains. Setup for a new task needs only modest resources.

Catalog record

CREATOROmar Naim, Krish Sharma, Nicholas Asher
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22767

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination Omar Naim, Krish Sharma, Nicholas Asher Abstract In this paper we introduce Tale , Task-Aware Layer Elimination, an inference-time algorithm that prunes entire transformer layers in an LLM by directly optimizing task-specific validation performance. We evaluate Tale on 9 tasks and 5 models, including LLaMA 3.1 8B, Qwen 2.5 7B, Qwen 2.5 0.5B, Mistral 7B, and Lucie 7B, under both zero-shot and few-shot settings. Unlike prior approaches, Tale requires no retraining and consistently improves accuracy while reducing computational cost across all benchmarks. Furthermore, applying Tale during finetuning leads to additional performance gains. Finally, Tale provides flexible user control over trade-offs between accuracy and efficiency. Mutual information analysis shows that certain layers act as bottlenecks, degrading task-relevant representations. Tale ’s selective layer removal remedies this problem, producing smaller, faster, and more accurate models that are also faster to fine-tune while offering new insights into transformer interpretability. 1 Introduction While Large Language Models (LLMs) have achieved remarkable success, they come with large computational overheads. This may prevent users in resource limited organizations or with high-throughput applications from using more capable models. This limitation has spawned an important research area on model compression. We focus here on the branch known as model pruning. Although it often reduces computational overhead, pruning typically requires complex implementation procedures, extensive retraining or fine-tuning, and may result in unpredictable performance degradation (Section 2 details related work). In contrast, having observed that not all layers contribute to a particular task, we propose a lightweight, greedy, iterative layer pruning algorithm, Task Aware Layer Elimination ( Tale ). Tale operates at inference time, is hardware agnostic, directly optimizes for task-specific accuracy at each pruning step and consistently offers improved results over the original model. This improvement persists in interactions with fine tuning on our tasks. As illustrated in Figure 1 and detailed in Section 3, Tale systematically evaluates all possible single-layer removals at each iteration, selecting the layer whose elimination results in the highest validation accuracy. This process continues iteratively until performance improvements fall below a predefined threshold, ensuring that only layers with minimal or negative impact on task performance are removed. Tale improves task specific accuracy and provides moderate computational reductions with minimal implementation complexity. This can be extremely useful for agentic systems with LLMs for dedicated tasks; TALE will make those LLMs perform better and more quickly. Our layer-wise pruning leverages the modular nature of transformer architectures, where each layer performs a complete transformation of the input representation through attention and feedforward mechanisms. This architectural property enables the removal of entire layers without requiring modifications to the remaining network structure. We evaluate Tale with five LLMs, LLaMA 3.1 8B, Qwen 2.5 7B, Qwen 2.5 0.5B, Mistral 7B and Lucie 7B, on 9 diverse benchmark datasets (Sections 4 and 5) both in zero-shot and few-shot settings. We also investigated interactions between pruning with Tale finetuning. We find consistent improvements in both accuracy and computational efficiency in all cases. This suggests pre-trained LLMs contain redundant or even detrimental layers for a given downstream task. We analyze layer flow using the notion of mutual information (MI) to support this hypothesis (Section 6). Additionally, our experiments show Tale ’s potential as a tool for understanding layer function in and across models (Section 7), thus aiding model interpretability. 2 Related Work Zhu et al. 2024 distinguishes four primary approaches to model compression: model pruning, quantization, low-rank approximation, and knowledge distillation. Our work focuses on pruning, which can be categorized into unstructured, structured, and semi-structured methods. Unstructured pruning removes individual parameters, resulting in irregular, sparse structures Han et al. 2015b ; Chen et al. 2015 ; Srinivas & Babu 2015 ; structured pruning eliminates entire components such as neurons, attention heads, or layers while maintaining the overall network structure He et al. 2017 ; Voita et al. 2019 ; Lagunas et al. 2021 ; Men et al. 2024 . Semi-structured pruning combines fine-grained control with structural regularity, and has been explored in recent work Li et al. 2023 ; Frantar & Alistarh 2023b ; Sun et al. 2024 . Early pruning methods leveraged second-order information for structured pruning LeCun et al. 1989 ; Hassibi et al. 1993 , but the field has since shifted toward computationally simpler, magnitude-based approaches that prune parameters by importance scores Han et al. 2015a ; See et al. 2016 ; Narang et al. 2017 Model pruning has benefited from information-theory. Building on the information bottleneck view of deep networks, Tishby et al. 2000 ; Tishby & Zaslavsky 2015 conceptualize layers as filters that preserve task-relevant information while discarding irrelevant input. This has led to model pruning proposals based on mutual information (MI). Fan et al. 2021 propose a layer-wise strategy that leverages MI estimates to reduce hidden dimensionality in a top-down manner. MI-guided approaches demonstrate that information-theoretic criteria can balance representation quality with model compression ( Ganesh et al. 2020 ; Westphal et al. 2024 ) . A central challenge, however, is the difficulty of estimating MI. Neural estimators such as MINE Ishmael Belghazi et al. 2018 provide principled solutions for high-dimensional settings, but in practice, probing classifiers Belinkov 2022 remain the dominant tool due to their efficiency and interpretability. With regards to large transformers, Zhang & Papyan 2025 proposes a pruning strategy based on matrix approximations. Xia et al. 2023 demonstrates that pruning 7B models into 1.3B and 2.7B variants via structured layer and hidden-dimension pruning, combined with dynamic batch loading for fine-tuning, can yield submodels outperforming comparably sized models trained from scratch, though not matching the original model. Complementary work Frantar & Alistarh 2023a ; Zhang et al. shows that contiguous blocks, especially attention layers, can often be pruned with limited performance loss, although critical “cornerstone” layers Zhang et al. remain essential for reasoning. Kim et al. 2024 investigates lock-level pruning based on weight importance, but these methods typically require fine-tuning techniques to recover baseline accuracy. Most pruning methods compress models while attempting to match the original performance but often suffer accuracy degradation, which requires retraining or fine-tuning Xia et al. 2024 . Improvements are generally measured relative to small models rather than original unpruned baselines. To the best of our knowledge, Tale is the only pruning method reported to improve over the original model’s performance without requiring any additional training or fine-tuning. Figure 1: Illustration of Tale layer elimination. Candidate layers (yellow) are tested for removal, and the best-performing ones above the threshold are permanently dropped (red) until no further improvement is possible. 3 Tale , our Greedy-Selection Algorithm 3.1 Basics and Intuitions A transformer maps a sequence of input vectors (x_{1},\cdots,x_{n}) to a corresponding sequence of output vectors through a stack of L layers. Each layer \ell transforms the hidden representations X^{(\ell)}=(x_{1}^{(\ell)},\ldots,x_{n}^{(\ell)}) into X^{(\ell+1)} through attention and feedforward blocks, connected by residual pathways. Removing layer \ell from this pipeline simply redirects the flow such that X^{(\ell-1)}\rightarrow X^{(\ell+1)} , a property that makes the architecture naturally amenable to layer-wise pruning. Our initial intuition for Tale came from examining the behavior of partial forward passes. Let h^{(k)} denote the hidden representation after k layers. Instead of always decoding from the final representation h^{(\text{L})} , we projected intermediate representations h^{(k)}

Read the complete text