Towards Explainability of SLMs by investigating Token Level Activation

Sayantani Ghosh, Rajashik Datta, Amit Kumar Das, Amlan Chakrabarti · 2026

Read Towards Explainability of SLMs by investigating Token Level Activation by Sayantani Ghosh, Rajashik Datta, Amit Kumar Das, Amlan Chakrabarti (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22377.txt.

Introduction

Ghosh and colleagues offer a light tool for seeing how small language models work, token by token. Old attention tools tend to pick out tokens that matter for form but not for sense, like marks of punctuation. Their Activation Flow Network ranks tokens by Layer-8 hidden-state strength in BERT. Words with sense land in the high bucket and drive shifts. Tokens that just hold form add far less. Layer 8 thus acts as a semantic consolidation zone. The tool gives a clear rival to attention-based study.

Catalog record

CREATORSayantani Ghosh, Rajashik Datta, Amit Kumar Das, Amlan Chakrabarti
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33593
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22377

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Explainability of SLMs by investigating Token Level Activation Sayantani Ghosh, Rajashik Datta, Amit Kumar Das, Amlan Chakrabarti Abstract Transformer-based language models such as BERT having 110M+ parameters have revolutionized natural language understanding, yet their internal mechanisms remain largely opaque to researchers and practitioners. Traditional attention-based interpretability methods often emphasize structurally important but semantically weak tokens such as punctuation marks rather than meaningful semantic relationships. This work introduces a lightweight and model-agnostic framework for quantifying token-level representational importance using hidden-state activation strengths at Layer 8 of BERT. The proposed Activation Flow Network (AFN) framework computes Token Activation Strength using the L2 norm of Layer-8 hidden representations, enabling direct ranking of semantically salient tokens. The study further introduces a threshold-based activation bucket formulation that partitions tokens into HIGH-activation and LOW-activation groups using an empirical upper-quartile activation boundary. Experimental observations demonstrate that semantically meaningful content words consistently occupy the HIGH-activation bucket and dominate representational activation shifts, while structurally supportive tokens contribute comparatively less. The results suggest that Layer 8 acts as a critical semantic consolidation zone balancing structural and semantic information processing. By revealing how activation magnitudes concentrate around semantically informative tokens, this work provides an interpretable and computationally efficient alternative to attention-centric analysis, contributing toward transforming BERT from a “black box” into a more transparent “glass box” model for natural language understanding. keywords BERT, SLM, Natural Language Understanding, Activation Strength 1 Introduction Transformer models have experienced a history of breakthroughs in the past few years, especially with the introduction of attention mechanisms. Recent improvements display the dominant use of large-scale pre-training on unlabeled data followed by task-specific fine-tuning. Despite BERT’s establishment of new performance standards, its internal core mechanism remains opaque. Attention weights often lead to misleading results by highlighting unmeaningful tokens like punctuations. BERT excels at pattern recognition but lacks genuine comprehension of meaning, struggling with ambiguous statements and context-dependent reasoning. Recent model-specific interpretability studies hinder application across different transformer architectures. Studies on BERT usually focus on a combination of all layers, or on final layers, missing crucial intermediate processing dynamics. There is a lack of identification of tokens carrying the maximum representational strength within specific transformer layers. Existent methods struggle to quantify and rank relative importance of different tokens based on internal activation strengths. Traditional probing methods rely on fixed layer/token selections, limiting their ability to capture dynamic nature of error signals across different positions. Existing methods often require expensive gradient computations or multiple model runs, limiting practical deployment. There has been limited research on how BERT’s middle layers (particularly layers 6-9) process and consolidate linguistic information despite evidence they contain crucial semantic processing. There is limited exploration of treating full activation tensors as structured data objects similar to images, missing opportunities for efficient processing techniques and underutilization of the natural sequential structure in both layer and token dimensions for comprehensive activation analysis. This research proposes a novel method by introducing a systematic approach for quantifying and analysing activation strengths of individual tokens at Layer 8 of the hidden layers of BERT, specifically on the bert-base-uncased model 3 . This choice uncovers the nuanced representational roles played by middle layers, often overlooked in favor of attention heads or classification outputs. We aim to identify which tokens are most salient or strongly encoded at this layer and explore patterns across a dataset of sentiment-bearing sentences. Unlike approaches focused only on attention weights or probing accuracy, this study demonstrates that hidden-state norms can directly reveal representational priorities, offering a lightweight and model-agnostic interpretability signal. The development of the Transformer architecture and its application in Bidirectional Encoder Representations from Transformers (BERT) 3 initiated a new era in Natural Language Processing (NLP), delivering significant performance gains across numerous benchmarks. However, this success introduced a critical challenge: the interpretability crisis. Our research builds directly upon three interconnected strands of prior work: the limitations of the attention mechanism, the layer-wise specialization within BERT, and the nascent efforts to use activation magnitudes as an alternative interpretability signal. 1.1 The Scrutiny of Attention: Moving Beyond the “What” to the “How” The very component that defined the Transformer—the attention mechanism—quickly became the subject of intense scrutiny as researchers sought to understand why these models were so effective. While attention weights were initially heralded as a path to explainability, showing which tokens the model was ”looking at,” studies soon revealed a critical flaw: attention weights are not explanations 6 . Researchers demonstrated that these weights often highlight tokens that are structurally important but semantically unmeaningful, such as punctuation and common function words, leading to misleading or contradictory interpretations. The opacity of BERT was reinforced, as the primary tool for ”looking inside” was deemed unreliable for genuine semantic understanding. This established the foundational need stated in our Introduction: to find an efficient and reliable method that focuses on semantic relationships and representational strength, rather than mere correlational attention. 1.2 Unpacking BERT: The Layer-Wise Linguistic Hierarchy As the interpretability community recognized the limitations of solely relying on attention, focus shifted to probing the information encoded in the high-dimensional hidden state vectors across different layers. Foundational work in this area, particularly on ”What Does BERT Learn about the Structure of Language?” 4 , confirmed that the 12-layer architecture is not a monolithic black box but a carefully constructed linguistic pipeline. This research established a hierarchy: • Lower Layers (1-5): Primarily encode surface features (e.g., Part-of-Speech, morphological information). • Middle Layers (6-9): Begin the complex task of consolidating syntactic and local semantic relationships (e.g., subject-verb agreement, phrase-level meaning). • Upper Layers (10-12): Specialize in abstract semantic features and task-specific classification preparation. This layer-wise division directly inspired our pragmatic choice of Layer 8. By focusing on Layer 8, we intentionally target the critical transition zone where structural information gives way to semantic consolidation, allowing us to observe how BERT balances these two forms of linguistic information before the final abstract layers, a crucial processing dynamic often missed by studies focusing only on final layers 5 . 1.3 The Shift to Activation Magnitude: A Lightweight Interpretability Signal Traditional probing methods often rely on training auxiliary classifiers or calculating computationally expensive gradient-based attribution scores (e.g., Integrated Gradients), which limits their practical deployment. This created a demand for a lightweight, model-agnostic interpretability signal. Researchers began to investigate the raw magnitude of the hidden state vectors, finding that the norms of these activations directly relate to component importance. This includes the observation and study of “massive activations”—scalar values within the hidden states that are orders of magnitude larger than typical activations and which are considered critical for model functionality 2 . More recent literature explicitly leverages hidden-state norms as a core feature for layer-wise activation analysis to classify layer specialization, demonstrating that vector magnitude can directly reveal a token’s representational priority 1 . 1.4 Our Contribution: The Activation Flow Network at Layer 8 Our research represents a synthesis of findings. We directly address the limitations identified in Section 1.1 by rejecting attention weights in favor of a magnitude-based approach. We operationalize the structural insights from Section 1.2 by specifically isolating the critical Layer 8. Finally, we adopt and formalize the lightweight methodology suggested in Section 1.3 by defining the L2 Norm of the Layer 8 hidden state as

Read the complete text