Does Self-Attention Need Separate Weights in Transformers?

Md Kowsher, Nusrat Jahan Prottasha, Chun-Nam Yu, Ozlem Ozmen Garibay, Niloofar Yousefi · 2024

Read Does Self-Attention Need Separate Weights in Transformers? by Md Kowsher, Nusrat Jahan Prottasha, Chun-Nam Yu, Ozlem Ozmen Garibay, Niloofar Yousefi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00359.txt.

Introduction

Md Kowsher and co-authors ask if self attention in transformers needs three apart weight sets, in a 2024 paper. Self attention lets a model track long range links in text, but it is heavy to train. Their shared weight form learns one matrix for the query, key, and value roles instead of three.

That cut the attention block's parameters by about two thirds and training time by about nine tenths. On small tasks of a standard language benchmark the shared model beat the baseline. Gains were clearest against two rival attention forms. The shared model also held up better on noisy and out of domain data, a sign of stronger general skill.

Catalog record

CREATORMd Kowsher, Nusrat Jahan Prottasha, Chun-Nam Yu, Ozlem Ozmen Garibay, Niloofar Yousefi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00359

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Does Self-Attention Need Separate Weights in Transformers? Md Kowsher, Nusrat Jahan Prottasha, Chun-Nam Yu, Ozlem Ozmen Garibay, Niloofar Yousefi Abstract Self-attention has revolutionized natural language processing by capturing long-range dependencies and improving context understanding. However, it comes with high computational costs and struggles with sequential data’s inherent directionality. This paper investigates and presents a simplified approach called "shared weight self-attention," where a single weight matrix is used for Keys, Queries, and Values instead of separate matrices for each. This approach cuts training parameters by more than half and significantly reduces training time. Our method not only improves efficiency but also achieves strong performance on tasks from the GLUE benchmark, even outperforming the standard BERT baseline in handling noisy and out-of-domain data. Experimental results show a 66.53% reduction in parameter size within the attention block and competitive accuracy improvements of 3.55% and 0.89% over symmetric and pairwise attention-based BERT models, respectively. 1 Introduction Natural language processing (NLP) has seen remarkable progress with the advent of transformer-based architectures Gillioz et al. 2020 ; Kowsher et al. 2022 . These models have revolutionized tasks such as machine translation Lopez 2008 , language modeling Jozefowicz et al. 2016 , and question answering Allam and Haggag 2012 ; Kowsher et al. 2024 , achieving better accuracy and performance. Central to the success of these models is the self-attention mechanism Vaswani et al. 2017 ; Shaw et al. 2018 , which allows them to weigh the importance of different words in a sentence dynamically. Self-Attention’s main challenges include computational inefficiency with quadratic complexity, difficulty in handling long-term dependencies effectively, and the lack of inherent directionality in capturing sequential relationships. While the attention mechanism itself has been extensively investigated Bielik and Vechev 2020 ; Choromanski et al. 2020 ; Zhuang et al. 2023 ; Phan et al. 2021 , and improvements in computational complexity have been proposed Kitaev et al. 2020 ; Zhu et al. 2020 ; Xiao et al. 2022 , the primary method retains the same architecture in using separate trainable weight matrices to compute Keys, Queries, and Values, which leads to a high parameter count and significant complexity for computing attention. We would like to ask: “Do we need the three weight matrix representations of (Key, Query, Value) for learning self-attention scores?" To address this question, we revisit the concept of self-attention and propose a novel shared weight self-attention mechanism that employs a single weight matrix for all three representations to reduce the parameter size and the time and memory complexity. Our shared weight matrix enables the model to efficiently capture the essential features needed for understanding semantics without the overhead of managing multiple matrices. The shared matrix is a regularization to capture the common weights learned from each representation. This simplification reduces the model’s computational footprint, retains the ability to focus on relevant parts of the input data effectively, and enhances prediction generalization for noisy input and out-of-domain test data. In this work, we explore alternative compatibility functions within the self-attention mechanism of Transformer-based encoder models, particularly BERT Devlin et al. 2018 . By utilizing a shared representation for (Key, Query, Value), our approach achieves substantial improvements in efficiency while maintaining the model’s performance without any compromise on accuracy. Our contributions can be summarized as follows: • We introduce a new shared self-attention mechanism that employs a single weight matrix, W_{s} for (Key, Query, Value). • Shared weight shows a 66.53% reduction in self-attention block parameters and 12.94% reduction in total BERT model parameters while maintaining performance across various downstream tasks. 2 Shared Self-Attention Figure 1: Comparison of traditional self-attention (left) and shared weight self-attention (right). 2.1 Preliminaries Consider an input matrix X\in\mathbb{R}^{n\times d} , where n is the sequence length and d is the dimensionality of the input space. The self-attention mechanism traditionally maps this input into three distinct representations: keys K , queries Q , and values V , using separate linear transformations with weight matrices W_{k} , W_{q} , and W_{v} respectively. We propose a unified representation using a single matrix W_{s} from which these mappings are derived, leading to a reduction in the number of parameters and accelerating the self-attention layer. 2.2 Self-Attention In traditional self-attention, distinct linear transformations are employed to generate keys K , queries Q , and values V from the input X . This process can be mathematically expressed as: K=XW_{k},\quad Q=XW_{q},\quad V=XW_{v}, where W_{k},W_{q},W_{v}\in\mathbb{R}^{d\times d} are learnable weight matrices corresponding to keys, queries, and values respectively. These matrices allow the model to adaptively focus on different parts of the input by calculating attention weights through the softmax-normalized dot product of queries and keys: \text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)V, (1) where d is the dimension of the model, which aids in stabilizing the learning process. 2.3 Shared Weight Self-Attention We define a shared transformation function \mathbf{S}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} parameterized by a weight matrix W_{s} containing learnable parameters: S=\mathbf{S}(X)=XW_{s},\quad W_{s}\in\mathbb{R}^{d\times d} This function \mathbf{S} is designed to capture the core semantic features of the input in a single compact representation S . To derive the keys, queries, and values vectors from the unified representation S , we introduce three separate diagonal transformation matrices D_{k},D_{q},D_{v} , each in \mathbb{R}^{d\times d} . These diagonal matrices act as element-wise scaling factors that adapt the shared representation S for specific roles in the attention mechanism: \displaystyle Q \displaystyle=SD_{q}=XW_{s}D_{q} \displaystyle K \displaystyle=SD_{k}=XW_{s}D_{k} \displaystyle V \displaystyle=SD_{v}=XW_{s}D_{v} This can be interpreted as having a special factorization of the weight matrices W_{q},W_{k},W_{v} used in standard attention as W_{q}=W_{s}D_{q} , W_{k}=W_{s}D_{k} , and W_{v}=W_{s}D_{v} , where W_{s} is shared and the diagonal D_{q},D_{k},D_{v} reduce the parameter count and allow for efficient and differentiated modulation of the base representation S . Now, we can calculate the attention score by following Equation 1 . 2.4 Experiments To evaluate the shared weight self-attention, we first pre-train the BERT model using shared weight self-attention. Subsequently, we assess the pre-trained BERT model across a range of NLP tasks, including the General Language Understanding Evaluation (GLUE) Benchmark Wang et al. 2018 and question-answering datasets such as SQuAD v1.1 Rajpurkar et al. 2016 and SQuAD v1.2 Rajpurkar et al. 2018 . For our baseline comparison, we use the standard self-attention-based BERT model Devlin et al. 2018 , as well as the symmetric and pairwise-based self-attention in BERT models from Courtois et al. 2024 . 2.5 Pre-training Shared Attention Based BERT Dataset: To pre-train the shared weight attention-based BERT model, we utilized the same corpora as the standard BERT-base-uncased model, specifically the BooksCorpus (800M words) Zhu et al. 2015 and English Wikipedia (2,500M words), resulting in a total of approximately 3.2 billion tokens. Pre-training Setup: We adopt the configuration settings of the standard BERT model Devlin et al. 2018 , which includes 12 layers, 768 hidden dimensions, and 12 attention heads. The maximum sequence length is set at 512 tokens. Regarding hyperparameters, we maintained the hidden dropout and attention dropout rates at 0.1. The pre-training is conducted over 20 epochs. We employ four H100 GPUs for computational resources, configuring each with a batch size of 132. The Adam optimizer Kingma and Ba 2014 was used, incorporating weight decay with \beta_{1}=0.9 and \beta_{2}=0.999 . Masked language modeling is performed using a mask ratio of 0.15. Pre-training Results Figure 2 presents

Read the complete text