Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models

Xin Liu, Qiyang Song, Qihang Zhou, Haichao Du, Shaowen Xu, Wenbo Jiang, Weijuan Zhang, Xiaoqi Jia · 2025

Read Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models by Xin Liu, Qiyang Song, Qihang Zhou, Haichao Du, Shaowen Xu, Wenbo Jiang, Weijuan Zhang, Xiaoqi Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07498.txt.

Introduction

Xin Liu and colleagues hunt for the attention heads that carry language skill inside multilingual models. Large models handle many languages, but which internal parts do that work stayed murky. Attention heads are a natural suspect because they steer what the model focuses on.

Their scoring method, LAHIS, rates each head with one forward and one backward pass. Applied to three open models, it finds heads tied to single languages and heads shared across many. Knowing which heads matter guides tuning and pruning without harming language coverage. The study turns a vague ability into located, testable parts that engineers can protect or improve.

Catalog record

CREATORXin Liu, Qiyang Song, Qihang Zhou, Haichao Du, Shaowen Xu, Wenbo Jiang, Weijuan Zhang, Xiaoqi Jia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07498

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Models Xin Liu, Qiyang Song, Qihang Zhou, Haichao Du, Shaowen Xu, Wenbo Jiang, Weijuan Zhang, Xiaoqi Jia Abstract Large language models (LLMs) increasingly support multilingual understanding and generation. Meanwhile, efforts to interpret their internal mechanisms have emerged, offering insights to enhance multilingual performance. While multi-head self-attention (MHA) has proven critical in many areas, its role in multilingual capabilities remains underexplored. In this work, we study the contribution of MHA in supporting multilingual processing in LLMs. We propose Language Attention Head Importance Scores (LAHIS), an effective and efficient method that identifies attention head importance for multilingual capabilities via a single forward and backward pass through the LLM. Applying LAHIS to Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, we reveal the existence of both language-specific and language-general heads. Language-specific heads enable cross-lingual attention transfer to guide the model toward target language contexts and mitigate off-target language generation issue, contributing to addressing challenges in multilingual LLMs. We also introduce a lightweight adaptation that learns a soft head mask to modulate attention outputs over language heads, requiring only 20 tunable parameters to improve XQuAD accuracy. Overall, our work enhances both the interpretability and multilingual capabilities of LLMs from the perspective of MHA. Code — https://github.com/Linuxin-xxx/LAHIS 1 Introduction Large language models (LLMs) have substantially advanced natural language processing (NLP). With their increasing pretraining on multilingual corpora, these models now demonstrate impressive capabilities in understanding and generating text across diverse languages ( Huang et al. 2025 ; Qin et al. 2024 ) . As a result, enhancing and analyzing their multilingual abilities has emerged as a central research objective ( Muennighoff et al. 2023 ; Qi, Fernández, and Bisazza 2023 ; CONNEAU and Lample 2019 ) . Alongside performance improvements, there is a growing interest in uncovering the internal mechanisms that enable multilingual processing in LLMs. Gaining a deeper understanding of these mechanisms can facilitate the development of more interpretable and controllable models. Recent studies have made progress on this area by identifying language-specific neurons or analyzing the function of each layer ( Zhao et al. 2024 ; Tang et al. 2024 ; Zhang et al. 2024 ; Wang et al. 2025 ; Zhong et al. 2024 ; Wendler et al. 2024 ) . However, one fundamental component of the Transformer architecture, multi-head self-attention (MHA), remains relatively underexplored in this context. In other domains, studies have identified attention heads that perform specialized functions, such as induction heads ( Olsson et al. 2022 ) , retrieval heads ( Wu et al. 2024 ) , successor heads ( Gould et al. 2023 ) , and safety heads ( Zhou et al. 2025 ) . These roles indicate that attention heads can exhibit functional specialization. Motivated by this, we hypothesize that multilingual LLMs may also contain attention heads that are selectively specialized for processing particular languages. In this paper, we investigate the contribution of individual attention heads within MHA to the multilingual capabilities of LLMs. To this end, we propose Language Attention Head Importance Scores (LAHIS) , an efficient method for attributing head importance with respect to multilingual performance. LAHIS utilizes a soft mask, implemented as a trainable matrix shaped to match the model’s attention structure, to estimate head importance by tracking its updates during training. This process requires only a single forward and backward pass over a target-language corpus. By applying LAHIS to Aya-23-8B, Llama-3.2-3B, and Mistral-7B-v0.1, we obtain attention head importance matrices across different languages and identify language heads that contribute significantly to multilingual capabilities. We define those that are highly important for specific languages as language-specific heads , and empirically validate both their effectiveness and specialization. Furthermore, we identify a set of language-general heads that consistently receive high importance scores across all languages. Language-specific heads can help address challenges faced by multilingual LLMs. In real-world applications such as dialogue systems and retrieval-augmented generation, multilingual input contexts are common. However, when conflicting facts are presented in different languages, the model’s information selection can become unpredictable and biased ( Park and Lee 2025 ) . Our experiments demonstrate that by enhancing or suppressing language-specific heads, we can steer the model’s attention and shift its preference toward the target-language context, thereby enabling more reliable and controllable cross-lingual reasoning. We further demonstrate that modulating language-specific heads provides a practical mechanism to mitigate off-target language generation ( Sennrich, Vamvas, and Mohammadshahi 2024 ) , a common issue where multilingual models are likely to output high-resource languages like English, even when the input is in a different language. By suppressing English-heads during generation, we effectively realign the output language with the input. Our experiments on the XL-Sum dataset show that this targeted intervention restores language fidelity while maintaining high summarization quality, revealing a controllable axis for correcting multilingual generation asymmetries in LLMs. Moreover, building on the discovery of language heads, we propose a lightweight adaptation method that enhances multilingual performance by training a small matrix aligned with the attention structure of the LLM, without modifying the LLM’s original weights. This trainable matrix updates only a few parameters associated with the identified heads and is inserted into the LLM as a gate to modulate the corresponding attention outputs during training or inference. Experiments on the XQuAD dataset across three models show that this adaptation, tuning only 14–20 parameters, yields an average accuracy improvement of 5 percentage points over the vanilla models, and outperforms the random-heads baseline by 4 points. Our key contributions are as follows: 1. We propose LAHIS, a lightweight and efficient method for estimating attention head importance with respect to multilingual capabilities. Applied to three LLMs across diverse languages, LAHIS reveals the existence of language-specific heads critical to individual language performance, and language-general heads beneficial across languages. 2. We show that language-specific heads offer a mechanism to control model behavior in multilingual contexts. By enhancing or suppressing these heads, we steer the model’s attention toward the target language, enhancing cross-lingual reasoning and reducing bias from conflicting multilingual inputs. 3. We demonstrate that modulating language-specific heads effectively mitigates off-target language generation. Our experiments show that suppressing English-heads realigns the outputs with input languages while maintaining generation quality. 4. We propose a lightweight adaptation that enhances multilingual performance by inserting a simple, attention-aligned trainable matrix to modulate certain language heads outputs. Tuning only 14–20 parameters, it improves XQuAD accuracy by 5 points on average and surpasses the random-heads baseline by 4 points. 2 Method 2.1 Preliminary In this section, we briefly introduce the Transformer architecture and the multi-head self-attention (MHA) mechanism ( Vaswani et al. 2017 ) , as they form the foundation for the rest of the paper. We mainly focus on the decoder-only transformer architecture, which is widely adopted in modern LLMs. This architecture generally consists of multiple layers, each comprising a multi-head self-attention module and a feed-forward network. The self-attention mechanism computes dependencies between tokens in an input sequence, assigning different weights to enable the LLMs to focus on the most relevant information. Multi-head self-attention further splits internal representations into multiple subspaces, allowing the models to capture diverse semantic features that a single head may struggle to represent effectively ( Vaswani et al. 2017 ) . Each head in the

Read the complete text