Mixture of Attention Schemes (MoAS): Learning to Route Between MHA, GQA, and MQA

Esmail Gumaan · 2025

Read Mixture of Attention Schemes (MoAS): Learning to Route Between MHA, GQA, and MQA by Esmail Gumaan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20650.txt.

Introduction

Esmail Gumaan lets each token choose its own attention bargain. Full multi-head attention gives the best quality but inflates the key-value cache at inference. Multi-query and grouped-query attention shrink the cache by sharing keys and values, usually at some quality cost. A static choice forces one compromise on every token.

Mixture of Attention Schemes routes dynamically instead: a learned router picks among the three schemes per token. On WikiText-2, dynamic routing edges out a static mixture of the same schemes and stays competitive with the full multi-head benchmark. The architecture promises conditional compute savings, spending memory only where the router judges it worthwhile. For deployment-minded model designers, per-token attention choice is a new dial to turn.

Catalog record

CREATOREsmail Gumaan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE7 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20650

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mixture of Attention Schemes (MoAS): Learning to Route Between MHA, GQA, and MQA
Esmail Gumaan

Abstract
The choice of attention mechanism in Transformer models involves a critical trade-off between modeling quality and inference efficiency. Multi-Head Attention (MHA) offers the best quality but suffers from large Key-Value (KV) cache memory requirements during inference. Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) reduce memory usage but often at the cost of model performance. In this work, we propose Mixture of Attention Schemes (MoAS), a novel architecture that dynamically selects the optimal attention scheme (MHA, GQA, or MQA) for each token via a learned router. We demonstrate that dynamic routing performs better than static averaging of schemes and achieves performance competitive with the MHA baseline while offering potential for conditional compute efficiency. Experimental results on WikiText-2 show that dynamic routing (val loss 2.3074) outperforms a static mixture (2.3093), validating the effectiveness of the proposed method. Our code is available at https://github.com/Esmail-ibraheem/Mixture-of-Attention-Schemes-MoAS .



1
Introduction

Large Language Models (LLMs) based on the Transformer architecture
[
13
]
have achieved remarkable success. However, their deployment is constrained by the memory required to store the Key-Value (KV) cache during autoregressive generation.

Standard Multi-Head Attention (MHA) maintains unique keys and values for every query head, resulting in a memory footprint that scales linearly with the number of heads. To mitigate this, Multi-Query Attention (MQA)
[
11
]
shares a single key-value head across all query heads, significantly reducing memory bandwidth and capacity requirements. Grouped-Query Attention (GQA)
[
1
]
interpolates between these extremes by grouping query heads to share key-value pairs.

While MQA and GQA offer efficiency gains, they generally underperform MHA in terms of perplexity and downstream task accuracy. We hypothesize that not all tokens require the full expressivity of MHA. Some tokens may be adequately processed with the approximated context of MQA, while others require the fine-grained relationships captured by MHA.

To address this, we introduce
Mixture of Attention Schemes (MoAS)
. Inspired by Mixture-of-Experts (MoE)
[
10
]
, MoAS employs a router to dynamically weight or select between MHA, MQA, and GQA branches for each token. This allows the model to learn an optimal balance between quality and efficiency.

2
Related Work

Efficient Transformers

Numerous works have attempted to reduce the quadratic complexity of self-attention. Sparse Transformers
[
4
]
and Longformer
[
2
]
introduce fixed sparse patterns. Linformer
[
14
]
and Reformer
[
7
]
utilize low-rank approximations and hashing, respectively. FlashAttention
[
5
]
optimizes memory access patterns for hardware efficiency.

Mixture of Experts

Conditional computation has been popularized by Mixture-of-Experts (MoE) models like the Switch Transformer
[
6
]
, which route tokens to different feed-forward networks. Recently, Mixture-of-Depths
[
9
]
proposed dynamically allocating compute by routing tokens around blocks entirely. MoAS extends this philosophy specifically to the attention mechanism’s internal structure.

KV Cache Optimization

As LLMs scale
[
3
,
12
]
, KV cache management becomes critical. PagedAttention
[
8
]
optimizes memory allocation. MQA
[
11
]
and GQA
[
1
]
structurally reduce the cache size. Our work builds directly on these structural innovations.

3
Method

3.1
Attention Schemes

We define three distinct attention variants as our ”experts”:

•

Type A: Multi-Head Attention (MHA)
:

H_{Q}=H_{KV}=H
. This is the standard mechanism with maximal expressivity.

•

Type B: Grouped-Query Attention (GQA)
:

H_{Q}=H
,
H_{KV}=G
, where
1<G<H
. This provides a middle ground. In our experiments, we use
G=2
for
H=6
.

•

Type C: Multi-Query Attention (MQA)
:

H_{Q}=H
,
H_{KV}=1
. This minimizes KV cache size but imposes the strongest bottleneck on the attention mechanism.

3.2
MoAS Architecture

Given an input token representation
x_{i}\in\mathbb{R}^{d}
, we compute the output of all schemes:

\displaystyle O_{MHA}

\displaystyle=\text{Attention}_{MHA}(x_{i})

(1)

\displaystyle O_{GQA}

\displaystyle=\text{Attention}_{GQA}(x_{i})

(2)

\displaystyle O_{MQA}

\displaystyle=\text{Attention}_{MQA}(x_{i})

(3)

3.2.1
Router

Everything is conditioned on a learned router that projects the input to a categorical distribution over the schemes:

r_{i}=W_{2}\cdot\text{GELU}(W_{1}x_{i})

(4)

g_{i}=\text{softmax}(r_{i})\in\mathbb{R}^{3}

(5)

where
W_{1}\in\mathbb{R}^{d/4\times d}
and
W_{2}\in\mathbb{R}^{3\times d/4}
form a lightweight Multi-Layer Perceptron (MLP).

The final output
y_{i}
for token
i
is the weighted sum:

y_{i}=\sum_{k\in\{MHA,GQA,MQA\}}g_{i,k}\cdot O_{k}

(6)

3.2.2
Load Balancing

To prevent the router from collapsing to a single scheme (e.g., always choosing MHA), we add an auxiliary load balancing loss:

\mathcal{L}_{balance}=\sum_{k}\left(\frac{1}{N}\sum_{i=1}^{N}g_{i,k}-\frac{1}{3}\right)^{2}

(7)

This encourages uniform usage of all attention types on average across the batch.

4
Experiments

4.1
Setup

We evaluate our method on the WikiText-2 language modeling benchmark. We train a decoder-only Transformer with the following specifications:

•

Layers: 4

•

Model Dimension (
d_{model}
): 384

•

Heads (
H
): 6

•

Block size: 256

•

Dropout: 0.1

We compare three models:

1.

Baseline MHA
: Standard Transformer with MHA.

2.

Static MoAS
: A static average of MHA, GQA, and MQA outputs (no routing).

3.

Dynamic MoAS
: The proposed method with learned routing.

All models are trained for 500 iterations with a batch size of 12 and learning rate
3\times 10^{-4}
.

4.2
Results

Table
1
presents the validation loss (perplexity-related metric) on WikiText-2.

Table 1:
Experimental Results on WikiText-2

Model

Parameters

Final Val Loss

Training Time (ms/iter)

Baseline MHA

7.19M

2.2940

\sim
500

Static MoAS

10.14M

2.3093

\sim
900

Dynamic MoAS

10.29M

2.3074

\sim
1680

The Baseline MHA achieves the lowest loss, which is expected given the small scale and absence of capacity constraints. However,
Dynamic MoAS outperforms Static MoAS
(2.3074 vs 2.3093), confirming that the router learns non-trivial routing policies that are superior to simple averaging.

The parameter count for MoAS variants is higher because we instantiate all three attention branches in parallel for this proof-of-concept. In a production inference scenario, one would only execute the selected branch(es).

5
Discussion & Conclusion

We proposed Mixture of Attention Schemes (MoAS), a method to dynamically route tokens between MHA, GQA, and MQA. Our experiments demonstrate that learned routing is effective and outperforms static mixing.

Read the complete text