Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models
Isaac Gerber · 2025
Read Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models by Isaac Gerber (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06633.txt.
Introduction
Isaac Gerber studies the feedforward networks inside transformer models in this 2025 paper. Attention gets most of the fame in these models. Each block also holds a two layer feedforward network with many parameters. Gerber tests how much that part matters during pretraining, not just in the finished model.
Experiments confirm the feedforward part is important to performance throughout training. A variant block with a three layer feedforward network was then tried. Using fewer such blocks beat the standard layout. It reached lower training loss with fewer total parameters and in less time. The result challenges the fixed two layer habit in transformer design. Builders of large language models may gain efficiency by rebalancing block internals.
Catalog record
| CREATOR | Isaac Gerber |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06633
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models Isaac Gerber Abstract Decoder-only transformer networks have become incredibly popular for language modeling tasks. State-of-the-art models can have over a hundred transformer blocks, containing billions of trainable parameters, and are trained on trillions of tokens of text. Each transformer block typically consists of a multi-head attention (MHA) mechanism and a two-layer fully connected feedforward network (FFN). In this paper, we examine the importance of the FFN during the model pre-training process through a series of experiments, confirming that the FFN is important to model performance. Furthermore, we show that models using a transformer block configuration with three-layer FFNs with fewer such blocks outperform the standard two-layer configuration delivering lower training loss with fewer total parameters in less time. 1 Introduction The decoder-only transformer architecture popularized by OpenAI’s GPT models [ 14 , 15 ] consists of many stacked transformer blocks that each include two key components: a self-attention mechanism, typically a form of multi-head attention (MHA), followed by a two-layer feedforward network (FFN). Although significant efforts have focused on optimizing the MHA, less attention has been paid to the FFN. A standard transformer block has more trainable parameters in the FFN than in the MHA, as explored in Section 2 . Given that top large language models (LLMs) have billions of trainable parameters and are pre-trained on trillions of tokens of text [ 11 ] , allocating a parameter budget efficiently is vital. On the other hand, pre-LLM research into multilayer perceptions (MLPs) show that FFNs with a hidden layer can act as universal function approximators [ 9 ] . For this reason, it is possible that adding a third, hidden layer to the transformer block FFNs will improve model performance. In this paper, we evaluate the importance of the FFN within the transformer block through a series of experiments, examining the performance of models with three linear layers, two linear layers (the baseline), one linear layer, and zero linear layers per block. We further determine if performance is driven by the FFN or the total number of trainable parameters, by changing the model dimension size ( d_{model} ) or the number of transformer blocks to keep the number of trainable parameters roughly constant. 2 Background Vaswani et al. [ 19 ] introduced the transformer model architecture, consisting of input embeddings, stacks of transformer blocks which contain an encoder and decoder, followed by output probabilities. Current approaches to transformer-based language models typically omit the encoder, stacking decoder-only transformer blocks [ 21 ] . The decoder transformer block has two main components: MHA and an FFN. The MHA mechanism learns three sets of weights, W^{q} , W^{k} , W^{v} , corresponding to the queries, key, and values, further split into h heads. Combined across all such heads, the dimensions of each weight matrix are d_{model}\times d_{model} , resulting in 3\times d_{model}\times d_{model} learnable weights. The standard FFN portion of the transformer block has two linear layers: a 4d_{model}\times d_{model} layer and a d_{model}\times 4d_{model} layer, resulting in 8\times d_{model}\times d_{model} learnable weights. When compared, we see that the standard transformer block learns parameters in an 8:3 ratio between the FFN and MHA, respectively. 3 Related Work Sukhbaatar et al. [ 18 ] show that complete removal of the linear layers and associated activation function significantly degrades model performance. However, their approach does not consider the inclusion of a single linear layer, nor does it examine model performance while retaining an activation function. He et al. [ 6 ] perform ablation studies on layers and full transformer blocks. However, their work only considers an already pre-trained model, Llama-2, and does not examine the effect on the pre-training process. [ 17 ] and [ 16 ] explore removing the intermediate layers between transformer blocks in the BERT model architecture, but do not explore the effect of the FFN within the transformer block itself. [ 20 ] studies the theoretical effect of transformer components, validating the results with experiments. They study the effect of the FFN width, finding that wider FFNs can lead to better model performance, but do not consider changing the number of layers in the FFN. [ 4 ] shows that the FFN in transformer models act as key-value memories, but do not study the effect of changing the number of layers on model performance. [ 13 ] examines replacing the block-specific FFNs in an encoder+decoder transformer model with shared FFNs, finding that FFNs can be shared across either the encoder or decoder or both. However, their experiments on decoder-only models are limited, and they look specifically at the machine translation task, not LLM pre-training. 4 Methods 4.1 Model Architectures and Implementation Details Figure 1: Transformer block architecture. A depicts a standard transformer block with two linear layers, B a transformer block with three linear layers, C a transformer block with a single linear layer, and D a transformer block with zero linear layers. The arrows at the bottom and top of each block indicate input and output from the previous and to the next sections of the models, respectively. In the linear layers, "d" refers to the model dimension size, d_{model} . A transformer model stacks many transformer blocks. Figure 1 shows the standard transformer block architecture and the three variants that we tested. In the one- and zero-layer variants, the dropout layer comes before the activation layer, whereas in the standard version, the dropout layer is the last. As dropout zeros out the output from some parameters during training, and activation functions map zero to zero, these are functionally equivalent. Our base model configuration contains two linear layers in the FFN, 24 transformer blocks in the overall model, and a d_{model} value of 1024, resulting in 323 million trainable parameters. We chose this base configuration to match the "GPT-3 Medium" architecture [ 2 ] . We considered variations with three linear layers, one linear layer, and zero linear layers. For each, we experimented with constant depth (number of transformer blocks) and d_{model} values, as well as varying either d_{model} or model depth to keep the parameter count within 3% of the baseline model. We used GELU [ 8 ] as our activation function and a dropout proportion of 0.1. The MHA layers have 16 heads. We used Xavier uniform initialization [ 5 ] for the MHA and linear layers, and normal initialization for the embedding layers. Our model uses the HuggingFace Tokenizer 1 1 1 https://huggingface.co/docs/transformers/en/main_classes/tokenizer with a maximum vocabulary of 10,000. We trained the models on a single epoch across each set of training data. We use the AdamW optimizer with a cosine learning rate decay with 300 warm-up steps with a maximum learning rate of 1.5^{-4}. We used a sequence length of 256 and 16 sequences per batch. Models were both trained and evaluated using one NVIDIA A100 in Google Colab 2 2 2 https://colab.research.google.com/ . 4.2 Data Sets We trained and evaluated our models on the "Booksum Complete Cleaned" 3 3 3 https://huggingface.co/datasets/ubaada/booksum-complete-cleaned a processed version of [ 10 ] (Booksum), available via the BSD-3 License, and "Wikitext 103 v1" [ 12 ] (Wikitext) data sets, available via Creative Commons Attribution-ShareAlike License.