Hardware-Centric Analysis of DeepSeek’s Multi-Head Latent Attention
Robin Geens, Marian Verhelst · 2025
Read Hardware-Centric Analysis of DeepSeek’s Multi-Head Latent Attention by Robin Geens, Marian Verhelst (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02523.txt.
Introduction
Robin Geens and Marian Verhelst analyze a memory-saving attention design from the hardware side in this 2025 paper. DeepSeek's latent attention squeezes key and value data into a compact space, shrinking the cache models must keep. How that change plays out on chips was unexamined.
The authors compare two ways to run the design, reusing stored projections or recomputing them, which trade memory traffic against arithmetic. Modeling across platforms shows the design shifts attention work toward the compute-bound side. It uses far less bandwidth than standard attention and stays steadier on bandwidth-poor hardware. Execution can be adapted to each chip's limits. The authors frame the design as a co-design opportunity for future AI accelerators.
Catalog record
| CREATOR | Robin Geens, Marian Verhelst |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02523
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hardware-Centric Analysis of DeepSeek’s Multi-Head Latent Attention Robin Geens, Marian Verhelst Abstract Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2, improves the efficiency of large language models by projecting query, key, and value tensors into a compact latent space. This architectural change reduces the KV-cache size and significantly lowers memory bandwidth demands, particularly in the autoregressive decode phase. This letter presents the first hardware-centric analysis of MLA, comparing it to conventional Multi-Head Attention (MHA) and evaluating its implications for accelerator performance. We identify two alternative execution schemes of MLA—reusing, resp. recomputing latent projection matrices—which offer distinct trade-offs between compute and memory access. Using the Stream design space exploration framework, we model their throughput and energy cost across a range of hardware platforms and find that MLA can shift attention workloads toward the compute-bound regime. Our results show that MLA not only reduces bandwidth usage but also enables adaptable execution strategies aligned with hardware constraints. Compared to MHA, it provides more stable and efficient performance, particularly on bandwidth-limited hardware platforms. These findings emphasize MLA’s relevance as a co-design opportunity for future AI accelerators. \orcid 0009-0000-2450-2660 \orcid 0000-0003-3495-9263 af1]MICAS, KU Leuven, Leuven, Belgium \corresp Email: [email protected] 1 Introduction DeepSeek-V3 [ 1 ] has been shown to significantly reduce training and inference costs compared to other commercial large language models, while maintaining competitive accuracy and usability. A key enabler of this efficiency is its use of Multi-Head Latent Attention (MLA), a novel attention mechanism where Q , K and V matrices are first projected into a low-dimensional latent space, and then projected into a higher-dimensional space to compute the attention scores. This approach allows for compact storage of the KV-cache entries during inference, drastically reducing memory bandwidth requirements, particularly in the decode stage. This letter presents a hardware-centric analysis of MLA’s decode-phase behavior on modern accelerators, comparing its performance to that of traditional Multi-Head Attention (MHA). The analysis quantifies the associated throughput and energy cost improvements, and evaluates the resulting shift in architectural requirements for efficient deployment. Although previous works have detailed the benefits of MLA as an algorithmic technique [ 2 , 3 , 4 ] , to the best of our knowledge, this is the first study of its kind to analyze the computational footprint and practical implications on hardware acceleration systems. Figure 1: Architecture of MHA and MLA. 2 Organization This letter begins with a review of standard MHA and the key modifications introduced in MLA. We then analyze the ordering of matrix multiplications in MLA, identifying trade-offs between compute and memory access. Building on these insights, we compare operation counts, memory access patterns, and algorithmic intensities of MHA and MLA. Finally, we model these characteristics across various hardware platforms using the Stream design space exploration framework to derive implications for accelerator architecture design. 2.1 Multi-Head Attention MHA [ 5 ] , shown in Figure 1 (left) is defined as follows: \displaystyle\text{MHA}(X)=\text{Concat}(\text{head}_{1},\dots,\text{head}_{n_{h}})\;W^{O} where each attention head is computed as: \displaystyle\text{head}_{i}=\text{SoftMax}\left(\frac{Q_{i}K_{i}^{T}}{\sqrt{D_{QK}}}\right)V_{i}=S_{i}\;V_{i} \displaystyle Q=X\;W^{Q};\quad K=X\;W^{K};\quad V=X\;W^{V} \displaystyle X\in\mathbb{R}^{L\times D_{\text{model}}} \displaystyle W^{Q},W^{K}\in\mathbb{R}^{n_{h}\times D_{\text{model}}\times D_{QK}};\quad W^{V}\in\mathbb{R}^{n_{h}\times D_{\text{model}}\times D_{V}}\quad \displaystyle W^{O}\in\mathbb{R}^{n_{h}D_{V}\times D_{\text{model}}} Q_{i} , K_{i} and V_{i} are obtained from slicing the Q , K and V matrices into n_{h} equal parts in the D -dimension. During autoregressive inference, K and V matrices are cached and updated incrementally as each new token is generated. 2.2 Multi-Head Latent Attention In MLA [ 6 ] (Figure 1 (right)), inputs X are first projected into a smaller, latent space 1 1 1 For simplicity, Rotary Positional Embeddings (RoPE) [ 7 ] is omitted here. : \displaystyle Q_{l}=X\;W_{\text{down}}^{Q};\quad C_{KV,l}=X\;W_{\text{down}}^{KV} \displaystyle Q=Q_{l}\;W_{\text{up}}^{Q};\quad K=C_{KV,l}\;W_{\text{up}}^{K};\quad V=C_{KV,l}\;W_{\text{up}}^{V} where \displaystyle W_{\text{down}}^{Q}\in\mathbb{R}^{D_{\text{model}}\times D_{Q,l}}\quad W_{\text{down}}^{KV}\in\mathbb{R}^{D_{\text{model}}\times D_{KV,l}} \displaystyle W_{up}^{Q}\in\mathbb{R}^{n_{h}\times D_{Q,l}\times D_{QK}};\quad W_{\text{up}}^{K}\in\mathbb{R}^{n_{h}\times D_{KV,l}\times D_{QK}}; \displaystyle\quad W_{\text{up}}^{V}\in\mathbb{R}^{n_{h}\times D_{KV,l}\times D_{V}} The computational benefits of this approach are twofold, assuming D_{Q_{l}},D_{KV_{l}}\ll D_{QK} : 1) the C_{KV_{l}} matrix is cached instead of the large K and V matrices, significantly reducing the memory footprint and 2) the number of parameters in the projection weights is much smaller. Table 1: Parameters of DeepSeek-V3 [ 1 ] model and derived variants Parameter MLA MHA (derived) MHA (scaled) D_{\text{model}} 7168 7168 4363 n_{h} 128 128 128 D_{Q,l} 1536 - - D_{KV,l} 512 - - D_{QK} 128 128 77 D_{V} 128 128 77 #params in 1 attention layer 174M 470M 172M In this letter, we analyze MLA with the hyperparameter instantiations proposed in DeepSeek-V3 and given in Table 1 . To compare MLA with standard MHA, we propose two MHA baselines: one with equivalent internal dimensions yet larger number of parameters ( \texttt{MHA}_{\texttt{l}} ) and another with an equivalent parameter count ( \texttt{MHA}_{\texttt{s}} ). 2.3 Order of Multiplications Computing the attention scores requires a projection from the cache’s latent space to the K - and V -spaces. A naive implementation of MLA would up-project the entire cached latent history before computing attention. However, this can be avoided by reordering operations: \displaystyle Z=QK^{T} \displaystyle=(Q_{l}\;W_{\text{up}}^{Q})(C_{KV,l}\;W_{\text{up}}^{K})^{T} \displaystyle=Q_{l}\underset{\hbox to6.49pt{\vbox to6.49pt{\pgfpicture\makeatletter\hbox{\;\lower-3.24614pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke} \lxSVG@begingroup@{fill} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} { {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{{{}}}{{}}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{}{}{}{}{}{{}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 4.21 0 C 4.21 2.33 2.33 4.21 0 4.21 C -2.33 4.21 -4.21 2.33 -4.21 0 C -4.21 -2.33 -2.33 -4.21 0 -4.21 C 2.33 -4.21 4.21 -2.33 4.21 0 Z M 0 0}{fill:none} \lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-1.70142pt}{-1.6111pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope {{{}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}{}W_{\text{up}}^{Q}\underset{\hbox to6.49pt{\vbox to6.49pt{\pgfpicture\makeatletter\hbox{\;\lower-3.24614pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke} \lxSVG@begingroup@{fill} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} { {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{{{}}}{{}}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{}{}{}{}{}{{}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 4.21 0 C 4.21 2.33 2.33 4.21 0 4.21 C -2.33 4.21 -4.21 2.33 -4.21 0 C -4.21 -2.33 -2.33 -4.21 0 -4.21 C 2.33 -4.21 4.21 -2.33 4.21 0 Z M 0 0}{fill:none} \lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-1.70142pt}{-1.6111pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope {{{}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}{}W_{\text{up}}^{K,T}\underset{\hbox to6.49pt{\vbox to6.49pt{\pgfpicture\makeatletter\hbox{\;\lower-3.24614pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke} \lxSVG@begingroup@{fill} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} { {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{{{}}}{{}}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{}{}{}{}{}{{}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 4.21 0 C 4.21 2.33 2.33 4.21 0 4.21 C -2.33 4.21 -4.21 2.33 -4.21 0 C -4.21 -2.33 -2.33 -4.21 0 -4.21 C 2.33 -4.21 4.21 -2.33 4.21 0 Z M 0 0}{fill:none} \lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-1.70142pt}{-1.6111pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope {{{}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}{}C_{KV,l}^{T} The order in which these matrix multiplications are performed has a significant impact on efficiency. A naive strategy that computes the leftmost and rightmost products first ( 1→3→2 ) is suboptimal, as it requires up-projecting the entire latent KV-cache before performing attention in the high-dimensional embedding dimension. A left-to-right ordering ( 1→2→3 ) incrementally transforms Q_{l} first to the full space and then to the K_{l} -space, deferring the attention computation to the K_{l} -space and reducing compute and bandwidth costs. Another alternative is to compute the middle product first ( 2→1→3 ), which directly transforms Q_{l} into the K_{l} -space. The approach of first computing the composite matrix W_{\text{up}}^{Q}W_{\text{up}}^{K,T} is known as weight absorption [ 8 ] , and this computation order is referred to as \texttt{MLA}_{\texttt{rc}} in the remainder of this letter. As shown in Figure 2 , the \texttt{MLA}_{\texttt{rc}} ordering generally yields the best performance, particularly for long KV-caches and small batch sizes. Figure 2: Required number of operations for different computation orders of Q_{l}\;W_{\text{up}}^{Q}\;W_{\text{up}}^{K,T}\;C_{KV,l}^{T} in the DeepSeek-V3 decode phase. 1→2→3 indicates left-to-right multiplication. For typical and high sequence length scenarios, first recomputing the absorbed weight matrix and transforming Q_{l} to the KV,l -space results in the least amount of operations. A similar analysis can be made for the output projection: \displaystyle Y=S\;V\;W^{O}=S\;C_{KV,l}\;W_{\text{up}}^{V}\;W^{O} In this case, executing right-to-left is typically most efficient.