InhibiDistilbert: Knowledge Distillation for a ReLU and Addition-based Transformer
Tony Zhang, Rickard Brannvall · 2025
Read InhibiDistilbert: Knowledge Distillation for a ReLU and Addition-based Transformer by Tony Zhang, Rickard Brannvall (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15983.txt.
Introduction
Tony Zhang and Rickard Brannvall rebuild a small transformer with a cheaper attention style in a 2025 paper. Standard attention multiplies large matrices at every step. It also leans on the softmax function, which is costly to compute. Inhibitor attention takes a different path. It uses Manhattan distances between tokens instead. A simple ReLU cutoff replaces softmax, promising real energy savings.
The team tunes the new mechanism for faster training. Then they teach it by knowledge distillation inside a DistilBERT frame. The student model reached competitive scores on standard language benchmarks. It also held up well on sentiment analysis tasks. Readers get a glimpse of attention without the usual arithmetic bill.
Catalog record
| CREATOR | Tony Zhang, Rickard Brannvall |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15983
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
InhibiDistilbert: Knowledge Distillation for a ReLU and Addition-based Transformer Tony Zhang, Rickard Brannvall Abstract This work explores optimizing transformer-based language models by integrating model compression techniques with inhibitor attention, a novel alternative attention mechanism. Inhibitor attention employs Manhattan distances and ReLU activations instead of the matrix multiplications and softmax activation of the conventional scaled dot-product attention. This shift offers potential computational and energy savings while maintaining model effectiveness. We propose further adjustments to improve the inhibitor mechanism’s training efficiency and evaluate its performance on the DistilBERT architecture. Our knowledge distillation experiments indicate that the modified inhibitor transformer model can achieve competitive performance on standard NLP benchmarks, including General Language Understanding Evaluation (GLUE) and sentiment analysis tasks. Introduction. Transformer-based language models have revolutionized natural language processing (NLP), achieving state-of-the-art performance across a wide range of tasks, from machine translation to sentiment analysis [ 6 ] . However, the computational and energy demands of these models, particularly those arising from the self-attention mechanism, pose significant challenges for deployment in resource-constrained environments. Although highly effective, the self-attention mechanism relies heavily on matrix multiplications, which are computationally expensive and energy-intensive As the scale of transformer models continues to grow, so does their environmental impact, with studies estimating that training a single large model can emit as much carbon as five cars over their lifetimes [ 5 ] . This has spurred research into more efficient alternatives, including model compression techniques such as knowledge distillation [ 3 ] and alternative attention mechanisms, like ReLUFormer [ 4 ] or Linformer [ 8 ] . Another alternative is the inhibitor attention [ 2 ] , which was introduced as a means to avoid using the softmax function and matrix multiplications. The motivation for this work is driven by the potential advantages of inhibitor attention over conventional dot-product-based attention under low-bit precision quantization. Scaled dot-product attention relies on floating-point matrix multiplication and Softmax activations, which can become challenging when quantized, leading to precision loss. This work takes a first step towards inhibitor transformer model compression by demonstrating that it can be trained via knowledge distillation to perform well on NLP benchmark tasks. Conducted as a Master Thesis project during the fall of 2024, it faced several resource limitations (e.g., access to powerful GPUs). Therefore, it is presented here as a short paper workshop contribution to invite collaborators. Table 1: Experiment comparing a pre-trained conventional DistilBert with the Inhibitor alternative pre-trained by task-agnostic knowledge distillation. Each model was fine-tuned to the GLUE Benchmarks and the IMDB tasks. The performance on each test was averaged over three runs. Models GLUE IMDb Conv. DistilBERT 77.0 92.82 Inhibi.DistilBERT 74.5 92.81 Method The dot-product attention score of the conventional Transformer is replaced with the Manhattan distance according to S=\frac{QK^{T}}{\sqrt{d}}\quad\longrightarrow\quad Z_{ij}=\sum_{k}\frac{\gamma}{\sqrt{d}}|Q_{ik}-K_{jk}| (1) where Q,K,V are the query, key, and value matrices of [ 6 ] and d is the size of the latent dimension. The attention head output is then similarly replaced H=\mathrm{softmax}(S)V\quad\longrightarrow\quad H_{il}^{\prime}=\eta\sum_{j}\left((V_{jl}^{+}-\bar{Z}_{ij})^{+}+(V_{jl}^{-}+\bar{Z}_{ij})^{-}\right) (2) written with in the notation (x)^{+}=\max(x,0) and (x)^{-}=\min(x,0) for the positive and negative ReLU functions, respectrively. The attenuating effect for the Inhibitor is obtained by ReLU instead of conventional Softmax. Note how the attention score is applied separately to the positive and negative parts of V, thresholded by the inhibitor attentions score. To allow for further calibration of the inhibition effect, a shift is applied to the inhibition score by first centering the score and then adjusting it with a shifting parameter \delta \bar{Z}_{ij}=(Z_{ij}-\langle Z_{ij}\rangle_{j}-\delta)^{+} (3) where \langle Z_{ij}\rangle_{j} denotes that the mean is calculated over the axis corresponding to index j . The purpose is to control when values from V can pass through unmodified. Compared to the previous formulation of the Inhibitor Transformer [ 2 ] , this article introduces a new set of learnable scalar parameters for the inhibitor, \gamma , \eta , and \delta , which are specific to each attention head. Experiments Our experiments were based on the DistilBERT [ 3 ] paper, incorporating elements also from the TinyBERT [ 9 ] and Greedy Layer-Wise Training [ 1 ] papers. We conducted two experiments: 1) task-agnostic KD, and 2) task-specific KD. Each is described in more detail below. For the task-agnostic KD experiments, instead of using a full-sized BERT model as the teacher as in [ 3 ] , we used the smaller pre-trained DistilBERT model for computational convenience and simpler alignment. For the task-specific KD we used a BERT already fine-tuned to the GLUE task was used as teacher. In both cases, weights were initialized from the teacher model. Experiment 1. We transfer knowledge from a dot-product-based DistilBERT teacher model to an inhibitor-based DistilBERT student model using a task-agnostic knowledge distillation strategy : • Layerwise Training : Each layer was trained sequentially using Mean Squared Error (MSE) loss to align self-attention outputs. Only the query, key, and value matrices of the current layer were updated, while all other weights remained frozen. The layerwise training followed a bottom-up approach, freezing all layers except for the current one being trained. • Full-Layer Training : After layerwise alignment, all layers were unfrozen and trained together using MSE loss applied to hidden states to refine the representations. The initial phase involved layerwise training to align the contextual representations between the teacher and student models using 10% of the Wikitext-103 corpus. In this phase, all weights in the student model were frozen except for the weight matrices of the query, key, and value components in the current layer being trained. The Mean Squared Error (MSE) loss function was applied to align the context outputs of corresponding layers in both models. Each layer was trained iteratively from the bottom to the top layer (layer 0 to layer 5). After training a layer for two epochs, its weights were frozen, and the next layer in the sequence was unfrozen. Following the layerwise training, a full-layer training phase was conducted using 60% of the Wikipedia 20231101 corpus. In this phase, all layers in the student model were unfrozen, allowing parameter updates across the entire network. MSE loss was applied to the hidden states to align the hidden layer outputs between the teacher and student models. Once the task-agnostic knowledge distillation was completed, the final weights of the inhibitor DistilBERT model were stored and used as the foundation for fine-tuning to more specific NLP tasks. Experiment 2. We performed task-specific knowledge distillation using a fine-tuned BERT model on GLUE benchmark [ 7 ] tasks. The goal was to transfer task-specific knowledge to the inhibitor model by using the BERT model as a teacher. The loss components consisted of: • Soft Probability Distillation Loss : The distillation loss function uses the teacher model’s soft probabilities to encourage the student model to replicate the teacher’s predictions. • Hidden State Loss : To help guide the inner layers of the student model toward better alignment with the teacher’s representations. In the second experiment, we extended our initial task-agnostic distillation approach by implementing task-specific knowledge distillation with a fine-tuned BERT model on GLUE benchmark tasks. The objective was to transfer the specialized knowledge from the BERT model (acting as the teacher) to the inhibitor model (acting as the student). This process involved two main components: the