Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation
Nurkhan Laiyk, Gerard I. Gallego, Javier Ferrando, Fajri Koto · 2026
Read Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation by Nurkhan Laiyk, Gerard I. Gallego, Javier Ferrando, Fajri Koto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19678.txt.
Introduction
Laiyk, Gállego, Ferrando, and Koto study whether function vectors exhibit language-agnosticity, using machine translation as a case study. Function vectors are vector representations of tasks drawn from model activations in in-context learning. While prior work shows multilingual model representations can be language-agnostic, it remains unclear whether the same holds for function vectors. Across three decoder-only multilingual language models, translation function vectors from a single English-to-X direction transfer to other target languages, consistently improving the rank of correct translation tokens across multiple unseen languages.
Catalog record
| CREATOR | Nurkhan Laiyk, Gerard I. Gallego, Javier Ferrando, Fajri Koto |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27433 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.19678
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation Nurkhan Laiyk, Gerard I. Gallego, Javier Ferrando, Fajri Koto Abstract Function vectors (FVs) are vector representations of tasks extracted from model activations during in-context learning. While prior work has shown that multilingual model representations can be language-agnostic, it remains unclear whether the same holds for function vectors. We study whether FVs exhibit language-agnosticity, using machine translation as a case study. Across three decoder-only multilingual LLMs, we find that translation FVs extracted from a single English → \to X direction transfer to other target languages, consistently improving the rank of correct translation tokens across multiple unseen languages. Ablation results show that removing the FV degrades translation across languages with limited impact on unrelated tasks. We further show that base-model FVs transfer to instruction-tuned variants and partially generalize from word-level to sentence-level translation. 1 Introduction Large language models (LLMs) have become central to multilingual NLP, showing strong performance across a wide range of cross-lingual and multilingual tasks ( 3 ) . A growing body of mechanistic work has begun to ask how these models represent and process information across languages ( 18 ; 9 ) . One recurring finding is that many internal representations are language-agnostic : they encode task-relevant content, such as semantic meaning, independent of language identity, so that similar concepts in different languages are mapped to nearby states in activation space. This property has been documented at several levels of abstraction. At the representation level, work on multilingual BERT showed that substantial cross-lingual alignment can emerge without explicit parallel supervision, enabling zero-shot transfer across languages and scripts ( 17 ; 24 ) . More recent studies on autoregressive LLMs have confirmed that concept representations can be largely language-agnostic, with intermediate activations partially separating “what is being said” from “in which language” it is expressed ( 23 ; 7 ) . Beyond static representations, language-agnosticity has also been observed at the level of grammatical concepts ( 2 ) and at the level of circuits where prior work suggests that some internal mechanisms are shared across languages ( 8 ; 27 ) . A natural next step is to ask whether this language-agnosticity also holds for task-level representations such as function vectors. Function vectors ( 21 ) offer a direct way to test this: they are compact task representations, extracted from a few in-context demonstrations, that can be added to the residual stream to induce a target behavior. In this work, we investigate this question in the context of machine translation (MT). This task provides a useful testbed for studying language-agnosticity of function vectors because it requires both language understanding and language-specific generation, making it a priori unclear whether the underlying computation should be shared across target languages. Our main contributions are as follows: • We show that translation FVs extracted from a single English \to X direction exhibit language-agnostic behavior, promoting correct translations across multiple target languages. • We confirm the cross-lingual nature of this effect through high-gain token analysis and substantial overlap in the selected FV heads across translation directions. • We further test the robustness of this effect through ablation, transfer from base to instruction-tuned models, and partial generalization from word-level to sentence-level translation. Figure 1: Example of FV effects on the next-token distribution. For a single source word, we show the top tokens whose probability ( \Delta p ) increases the most after FV, highlighting that the largest gains often correspond to plausible translations across multiple languages. 2 Related work Multilingual language models. Multilingual language models are trained on text from many languages, often with deliberate choices about how training data are distributed across them. Some models use relatively balanced multilingual corpora in order to support cross-lingual transfer, including mT5 ( 25 ) , and Aya ( 22 ) . In contrast, models such as Llama-3 ( 11 ) and Gemma-2 ( 19 ) are trained on heavily English-dominant data, yet still exhibit strong multilingual capabilities. These different pretraining choices are often assumed to affect the emergence of multilingual and cross-lingual representations ( 5 ; 17 ; 1 ) . Language-agnosticity in multilingual LLMs. 7 provide causal evidence for language-agnostic concept representations in decoder-only LLMs using activation patching in word-to-word translation. They show that concept and language information can be manipulated independently, and that averaging concept representations across languages preserves or even improves translation performance, suggesting a shared conceptual space rather than fully language-specific encodings. These findings extend beyond lexical semantics to grammatical structure. 2 use sparse autoencoders to study morphosyntactic concepts and show that properties such as number, gender, and tense are often encoded in feature directions shared across many typologically diverse languages. Shared structure has also been identified at the circuit level. 8 analyze subject–verb agreement in English and Spanish and find that the two languages rely on highly similar circuits, including a language-independent “subject number” direction in the residual stream that transfers causally across languages through activation steering. 10 show that some recall mechanisms generalize across languages while others remain language-dependent. In decoder-only multilingual LLMs, the last-token representation behaves like a function vector that first encodes the relation to be retrieved and later composes in the language, whereas the final extraction step remains language-specific. Interpretability of machine translation in LLMs. Earlier work on understanding translation in neural models focused on observational analyses such as neuron activation patterns and representation probing ( 15 ; 20 ; 23 ) . More recent studies have moved toward causal, circuit-level analyses that identify the specific components responsible for translation behavior. 26 study word-level translation in decoder-only LLMs using subspace-intervened path patching. They find that translation behavior depends on a sparse subset of components, with attention heads and MLPs playing distinct functional roles. They further show that fine-tuning these translation-relevant components can improve translation performance while largely preserving general capabilities. 13 extend this line of work to sentence-level few-shot MT. Using activation patching on clean and corrupted prompts, they decompose MT into two separable subtasks: target language identification and sentence equivalence. They find that these subtasks are mediated by distinct sparse sets of attention heads across many translation directions. Based on these heads, they construct steering vectors for instruction-free MT and use ablations to show that the identified components are causally important. 3 Function Vectors Function vectors ( 21 ) are extracted from in-context learning (ICL) demonstrations and applied as an additive intervention in the residual stream. For each task t , we assume a dataset of ICL prompts P_{t} and a corrupted version \tilde{P}_{t} formed by shuffling demonstration labels. The procedure first selects a small set of attention heads using activation patching. For a given task t , it compares prompts from P_{t} (correct demonstrations) to corrupted prompts \tilde{p}\in\tilde{P}_{t} . For each head (\ell,j) , the head output on a corrupted prompt \tilde{p} is patched with the head’s mean output over p\in P_{t} , denoted \bar{a}^{t}_{\ell j} . The causal indirect effect (CIE) of head (\ell,j) is defined as the resulting change in the probability of the correct answer token y : \mathrm{CIE}_{\ell j}(\tilde{p})=P\!\left(y\mid\tilde{p};\;a_{\ell j}\leftarrow\bar{a}^{t}_{\ell j}\right)-P\!\left(y\mid\tilde{p}\right). Heads are ranked by their average indirect effect across corrupted prompts, and the top heads form the selected set A .