Language Surgery in Multilingual Large Language Models

Joanito Agili Lopo, Muhammad Ravi Shulthan Habibi, Tack Hwa Wong, Muhammad Ilham Ghozali, Fajri Koto, Genta Indra Winata, Peerat Limkonchotiwat, Alham Fikri Aji, Samuel Cahyawijaya · 2025

Read Language Surgery in Multilingual Large Language Models by Joanito Agili Lopo, Muhammad Ravi Shulthan Habibi, Tack Hwa Wong, Muhammad Ilham Ghozali, Fajri Koto, Genta Indra Winata, Peerat Limkonchotiwat, Alham Fikri Aji, Samuel Cahyawijaya (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12450.txt.

Introduction

Joanito Agili Lopo and co-authors study how large language models line up languages inside their own layers, and how that lining-up can be steered, in a 2025 paper. In the middle layers, the same idea in different languages seems to land in nearby spots, while the outer layers keep each language's flavor. The team confirms this pattern, measures how it behaves, and compares it with models built to align languages on purpose.

On that base they build a control method that injects a language signal at run time. It can force the reply's language with care while keeping the meaning intact. Their tests show it eases a stubborn flaw of big models: drifting into the wrong language mid-answer. The authors frame the work as both a map of multilingual internals and a practical steering wheel.

Catalog record

CREATORJoanito Agili Lopo, Muhammad Ravi Shulthan Habibi, Tack Hwa Wong, Muhammad Ilham Ghozali, Fajri Koto, Genta Indra Winata, Peerat Limkonchotiwat, Alham Fikri Aji, Samuel Cahyawijaya
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-12450

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Language Surgery in Multilingual Large Language Models Joanito Agili Lopo, Muhammad Ravi Shulthan Habibi, Tack Hwa Wong, Muhammad Ilham Ghozali, Fajri Koto, Genta Indra Winata, Peerat Limkonchotiwat, Alham Fikri Aji, Samuel Cahyawijaya Abstract Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across tasks and languages, revolutionizing natural language processing. This paper investigates the naturally emerging representation alignment in LLMs, particularly in the middle layers, and its implications for disentangling language-specific and language-agnostic information. We empirically confirm the existence of this alignment, analyze its behavior in comparison to explicitly designed alignment models, and demonstrate its potential for language-specific manipulation without semantic degradation. Building on these findings, we propose Inference-Time Language Control ( ITLC ), a novel method that leverages latent injection to enable precise cross-lingual language control and mitigate language confusion in LLMs. Our experiments highlight ITLC ’s strong cross-lingual control capabilities while preserving semantic integrity in target languages. Furthermore, we demonstrate its effectiveness in alleviating the cross-lingual language confusion problem, which persists even in current large-scale LLMs, leading to inconsistent language generation. This work advances our understanding of representation alignment in LLMs and introduces a practical solution for enhancing their cross-lingual performance. 1 1 1 Our code is published at https://github.com/SEACrowd/itlc . 1 Introduction Large Language Models (LLMs) have revolutionized natural language processing, demonstrating remarkable generalization capabilities across diverse tasks and languages ( Brown et al. 2020 ; Le Scao et al. 2023 ; Anil et al. 2023 ; Team et al. 2025 ; Cohere et al. 2025 ; Singh et al. 2025 ) . Their ability to adapt to new tasks in few-shot and even zero-shot settings highlights their efficiency and versatility ( Bang et al. 2023 ; Susanto et al. 2025 ) . Prior works have identified a naturally emerging representation alignment across layers in LLMs, particularly in the middle layers of LLMs ( Chang et al. 2022 ; Zhao et al. 2024a ) . This emerging alignment in LLMs is the key factor in their ability to handle multiple languages Cahyawijaya 2024 ; Tang et al. 2024 ; Wilie et al. 2025 , which is pivotal for their cross-lingual capabilities. However, several questions remain open, such as whether this emerging alignment behaves similarly to alignment in models trained with enforced alignment objectives ( Reimers and Gurevych 2020 ; Yang et al. 2019a ; Feng et al. 2022 ; Limkonchotiwat et al. 2022 ; Limkonchotiwat et al. 2024 ) , how this alignment can be utilized to further enhance LLMs, etc. Figure 1: We inspect the alignment in the middle layer representation of LLMs, allowing us to disentangle the language-specific and language-agnostic information. By exploiting this behavior, we are able to achieve Inference-Time Language Control ( ITLC ), alleviating the language confusion problem in LLMs. In this work, we investigate the phenomenon of representation alignment in LLMs, focusing on its occurrence, distinction, and potential applications. We aim to confirm the presence of this representation alignment and contrast it with alignment in LLMs with strictly designed alignment, such as multilingual SentenceBERT Reimers and Gurevych 2019 or LaBSE Feng et al. 2022 . Our findings highlight that, unlike LLMs with strictly designed alignment, the naturally emerging alignment in recent LLMs demonstrates a much stronger retention of language-specific information with \sim 30% performance drop compared to LLMs with strictly designed alignment, with almost >90% performance drops relative to other non-aligned layers. Upon further investigation, we found a potential method to disentangle language-specific and language-agnostic information in the aligned representation. By exploiting the disentangled language-specific and language-agnostic information, we demonstrate a simple but effective method to control the generation of language from such a representation, enabling us to achieve Inference-Time Language Control ( ITLC ) as showcased in Figure 1 . We demonstrate the effectiveness of ITLC in two downstream applications: 1) zero-shot cross-lingual language generation and 2) mitigating language confusion in LLMs Marchisio et al. 2024 . Our contribution in this work is fourfold: • We confirm the presence of representation alignment in LLMs, providing empirical evidence of this phenomenon (§ 3.2 ). • We contrast natural alignment with strictly designed alignment, highlighting their comparable impact on cross-lingual generalization while emphasizing their differences in alignment locations and the extent of language-specific information retention (§ 3.2 ). • We investigate a method to extract language-specific information from aligned representations, showcasing the potential for language-specific manipulation while preserving the semantic integrity of the generation (§ 4.1 ). • We introduce ITLC , a novel method that enables cross-lingual language control and mitigates language confusion problems that retain semantic integrity in target languages (§ 5 ). 2 Related Work 2.1 Representation Alignment in LLMs Representation alignment refers to the process by which semantically identical inputs expressed in different languages are mapped to similar internal embeddings within LLMs ( Park et al. 2024 ; Wu and Dredze 2020 ; Chang et al. 2022 ) . Originally, representation alignment is strictly embedded into the modeling objective to ensure output consistency across languages and to enable generalization in cross-lingual transfer tasks ( Pires et al. 2019 ; Wu and Dredze 2019 ; Reimers and Gurevych 2020 ; Feng et al. 2022 ; Choenni et al. 2024 ) . Several studies have observed a tendency for LLMs to align representations across different languages ( Wendler et al. 2024 ; Zhao et al. 2024b ; Mousi et al. 2024 ) . This is done by measuring the similarity between embeddings of parallel sentences across different languages ( Ham and Kim 2021 ; Gaschi et al. 2023 ; Cahyawijaya 2024 ) . Several benchmark datasets can be used for this purpose. Our work measure the degree of alignment across various layers between strictly and naturally aligned models to contrast the two and understand its relation to language-specific and language-agnostic capabilities ( Kulshreshtha et al. 2020 ; Libovický et al. 2020 ; Hua et al. 2024 ; Wilie et al. 2025 ) of LLMs. Figure 2: Cross-lingual similarity across different layers in LaBSE and Qwen2.5-0.5B. LaBSE exhibits high cross-lingual similarity in its final layer, whereas Qwen2.5-0.5B shows this similarity in the middle layer. This difference suggests that the alignment of representations occurs at distinct positions within the two models. 2.2 Latent Controllability in LLMs LLMs controllability is crucial for ensuring that the systems adhere with human intentions. Through mechanisms such as adapter Pfeiffer et al. 2020 ; Hu et al. 2022 , prompting Lin et al. 2021 ; Bai et al. 2022 , latent manipulation Madotto et al. 2020 ; Ansell et al. 2021 , etc, we aim to gain control over the behavior of LLMs. Various aspects have been explored in LLM controllability, including internal knowledge Madotto et al. 2020 ; Xu et al. 2022 , styles & personas Lin et al. 2021 ; Wagner and Ultes 2024 ; Cao 2024 , languages Üstün et al. 2020 ; Ansell et al. 2021 , human values Bai et al. 2022 ; Cahyawijaya et al. 2025a , etc. Recent works show that latent states in LLMs exhibit discernible patterns for distinguishing truthful outputs from hallucinated ones, suggesting an intrinsic awareness of fabrication ( Li et al. 2023 ; Duan et al. 2024 ; Ji et al. 2024 ; Chen et al.

Read the complete text