Attention Is All You Need For Mixture-of-Depths Routing

Advait Gadhikar, Souptik Majumdar, Niclas Popp Piyapat Saranrittichai, Martin Rapp, Lukas Schott · 2024

Read Attention Is All You Need For Mixture-of-Depths Routing by Advait Gadhikar, Souptik Majumdar, Niclas Popp Piyapat Saranrittichai, Martin Rapp, Lukas Schott (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20875.txt.

Description

Abstract Advancements in deep learning are driven by training models with increasingly larger numbers of parameters, which in turn heightens the computational demands. To address this issue, Mixture-of-Depths (MoD) models have been proposed to dynamically assign computations only to the most relevant parts of the inputs, thereby enabling the deployment of large-parameter models with high efficiency during inference and training.

Catalog record

CREATORAdvait Gadhikar, Souptik Majumdar, Niclas Popp Piyapat Saranrittichai, Martin Rapp, Lukas Schott
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20875

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention Is All You Need For Mixture-of-Depths Routing Advait Gadhikar, Souptik Majumdar, Niclas Popp Piyapat Saranrittichai, Martin Rapp, Lukas Schott Abstract Advancements in deep learning are driven by training models with increasingly larger numbers of parameters, which in turn heightens the computational demands. To address this issue, Mixture-of-Depths (MoD) models have been proposed to dynamically assign computations only to the most relevant parts of the inputs, thereby enabling the deployment of large-parameter models with high efficiency during inference and training. These MoD models utilize a routing mechanism to determine which tokens should be processed by a layer, or skipped. However, conventional MoD models employ additional network layers specifically for the routing which are difficult to train, and add complexity and deployment overhead to the model. In this paper, we introduce a novel attention-based routing mechanism A-MoD that leverages the existing attention map of the preceding layer for routing decisions within the current layer. Compared to standard routing, A-MoD allows for more efficient training as it introduces no additional trainable parameters and can be easily adapted from pre-trained transformer models. Furthermore, it can increase the performance of the MoD model. For instance, we observe up to 2 % 2\% higher accuracy on ImageNet compared to standard routing and isoFLOP ViT baselines. Furthermore, A-MoD improves the MoD training convergence, leading to up to 2 × 2\times faster transfer learning. 1 Introduction Increasing the model size has enabled transformer-based deep learning models to achieve state-of-the-art performance across various domains, including computer vision ( Dosovitskiy et al., 2021 ) and natural language processing ( Hoffmann et al., 2022 ; Kaplan et al., 2020 ) – even unlocking emergent capabilities ( Wei et al., 2022 ) . However, the computational costs of these large models present significant challenges ( Thompson et al., 2020 ) . Therefore, reaching a Pareto-optimal model to maximize both efficiency and performance is crucial (see Fig. 1 ). Jacobs et al. (1991) originally introduced conditional computation via mixture of experts, laying the foundations to increase model sizes while maintaining FLOPs, by dynamically activating only a subset of the model parameters, termed experts, conditioned on the input. This principle allowed scaling towards outrageously large networks ( Shazeer et al., 2016 ) and is leveraged at the forefront of current Large Language Models (LLMs) ( Jiang et al., 2024 ) . Figure 1: Accuracy vs FLOPs Pareto-curve for A-MoD in comparison with MoD and ISOFlop models on ImageNet-1k. Compared to standard deep learning models Dosovitskiy et al. (2021) ; Wang et al. (2024) ; He et al. (2016) , dynamic models have received less research attention and are often not yet competitive on the Pareto front of performance and runtime on standard GPU architectures. Here, we focus on further advancing the field of dynamic compute. Previous Block MoD Block MHSA Attn. Maps MLP Tokens Router MHSA MLP … Tokens Additional Trainable Layer(s) Routing Scores \sum Prev. Attention Maps Reduce Avg. Routing Scores or (a) MoD Model (b) Standard Routing (as in Raposo et al. (2024) ) (c) A-MoD (ours) Figure 2: MoD model (a) with standard routing (b) vs. our A-MoD attention routing (c). Recently, Raposo et al. (2024) introduced Mixture-of-Depths (MoD) as a variant of mixture of experts. In MoD models, the computational costs are dynamically reduced by processing only a subset of tokens in a layer while the remaining tokens skip the layer (see Fig. 2 a). Compared to baselines with equivalent FLOPs, MoDs can perform favorably on language tasks. A crucial component of MoD is its router, which receives tokens as inputs and, given a user-defined capacity, determines which tokens should enter or skip a layer. The router usually consists of a linear layer that is jointly trained along with the model ( Fig. 2 b). The routing mechanisms heavily influence the model performance for several reasons. First, routing introduces noise into the training process, as the routing is a discrete decision and is often performed at multiple layers and per token. Second, routers depend on additional layers, and hence, need to be trained from scratch when adapting a vanilla pretrained model to an MoD model. Lastly, the router adds a small computational overhead to the sparsified model. Hence, in this paper, we ask and address the question: Can we improve the routing mechanism in MoD models based on information that is already available within the model, instead of using additional trainable parameters within the router? We find the answer to our question in the attention mechanism of commonly used transformer architectures ( Vaswani et al., 2017 ; Dosovitskiy et al., 2021 ) . We assume that the attention maps can be used to estimate the importance of a token, by averaging its interaction with other tokens. Based on that, we propose to aggregate the information in the attention maps and use it as an importance measure for token routing in MoD. We call our method attention routing for MoD: A-MoD ( Fig. 2 c). We find that A-MoD can outperform standard routing in MoD networks across a range of model sizes and tasks consistently (as shown in Fig. 1 ). Not only is our A-MoD parameter-free, but it can also be applied to adapt off-the-shelf pretrained transformer models to MoDs with almost no additional training. We further validate our method empirically and show that routing scores computed by A-MoD are better correlated with token importance estimates compared to routing scores from standard routers. This paper presents a significant advancement in the application of MoD to the visual domain.Our primary contributions are: • We find that MoD is not only viable but also advantageous for visual tasks, providing empirical evidence that it can outperform traditional models in terms of both FLOPs and performance. • We introduce A-MoD , a parameter-free routing method for MoDs based on the attention maps to compute token importance. • We demonstrate that A-MoD outperforms a standard router across a range of datasets for MoDs on finetuning and transfer learning. In the case of transfer learning, A-MoD exhibits faster convergence of MoD models transferred from a dense pretrained model. • Compared to standard MoD, A-MoD consistently selects important tokens, and routing decisions correlate with leave-one-out token importance that is estimated by removing tokens. 2 Related Work Attention Maps The attention mechanism ( Bahdanau, 2014 ) enables models to learn long-range dependencies within sequences with a constant number of operations. Transformers ( Vaswani et al., 2017 ) leverage the attention mechanism in the language domain and have become a de facto standard model. Dosovitskiy et al. (2021) further adapted transformers to the vision domain by treating image patches as tokens, introducing the vision transformer (ViT). For images, attention maps have been shown to focus on key areas such as objects in the image ( Carion et al., 2020a ; Jetley et al., 2018 ) . In this paper, we utilize this property for effective routing of tokens within neural networks. Furthermore, we use the Data-efficient image Transformers DeiT-T and DeiT-S ( Touvron et al., 2021 ) instead of vanilla ViT-T and ViT-S models, as small ViTs do not generalize well when trained on smaller datasets ( Dosovitskiy et al., 2021 ) . Mixture of Experts and Mixture-of-Depths

Read the complete text