Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization

Iñigo Pikabea, Iñaki Lacunza, Oriol Pareras, Carlos Escolano, Aitor Gonzalez-Agirre, Javier Hernando, Marta Villegas · 2025

Read Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization by Iñigo Pikabea, Iñaki Lacunza, Oriol Pareras, Carlos Escolano, Aitor Gonzalez-Agirre, Javier Hernando, Marta Villegas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22577.txt.

Introduction

Inigo Pikabea and co-authors fix in 2025 a flaw in image text models that makes them reply in English no matter the tongue of the query. The flaw, named in past work, springs from thin multi tongue image training data. Their fix keeps the base model's tongue skills while it learns sight tasks.

Text only data in many tongues is fed in all through the sight tune span. Wide tests show tongue truth rose a lot with no loss in sight scores. A merge trick also lifted tongue truth but cost sight scores. The plain feed path gives scale and balance at once.

Catalog record

CREATORIñigo Pikabea, Iñaki Lacunza, Oriol Pareras, Carlos Escolano, Aitor Gonzalez-Agirre, Javier Hernando, Marta Villegas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22577

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization Iñigo Pikabea, Iñaki Lacunza, Oriol Pareras, Carlos Escolano, Aitor Gonzalez-Agirre, Javier Hernando, Marta Villegas Abstract Rapid advancements in Visual Language Models (VLMs) have transformed multimodal understanding but are often constrained by generating English responses regardless of the input language. This phenomenon has been termed as Image-induced Fidelity Loss (IFL) and stems from limited multimodal multilingual training data. To address this, we propose a continuous multilingual integration strategy that injects text-only multilingual data during visual instruction tuning, preserving the language model’s original multilingual capabilities. Extensive evaluations demonstrate that our approach significantly improves linguistic fidelity across languages without degradation in visual performance. We also explore model merging, which improves language fidelity but comes at the cost of visual performance. In contrast, our core method achieves robust multilingual alignment without trade-offs, offering a scalable and effective path to mitigating IFL for global VLM adoption. \caption@setdefaultlabelsep period 1 Introduction Figure 1: Language Fidelity (LF) accuracy on Crossmodal-3600. (BM: Base Model, TR: model trained with multilingual Textual Regularization, TR+M: TR and merging the final model with the original LLM Backbone) Large Language Models (LLMs) have significantly advanced multimodal understanding, leading to the rise of VLMs, which integrate vision encoders into LLM backbones. A widely adopted paradigm is the LLaVA-style architecture ( Liu et al., 2023b ; Liu et al., 2024a ) , where a decoder-only LLM is coupled with a vision encoder and an adapter module to align visual representations with textual embeddings. Despite their success, VLMs exhibit a strong bias toward English due to the predominance of monolingual vision-language training data. Consequently, they often generate English responses regardless of the input language, a phenomenon termed Image-induced Fidelity Loss (or IFL) ( Hinck et al., 2024 ) . This issue stems from limitations in the underlying LLM rather than the visual representations. Ensuring multilingual capability in VLMs is essential for their adoption across diverse linguistic communities, as reliance on English-centric outputs risk erasing cultural and linguistic nuances. Prior work ( Qiu et al., 2022 ; Li et al., 2023b ) has explored dataset translation, but this approach incurs high computational costs and introduces translation errors, especially in images with language-dependent elements. In this paper, we propose an alternative solution by integrating multilingual text-only data during the visual instruction tuning process. Additionally, we explore model merging, combining the visually fine-tuned model with the original multilingual backbone LLM to further preserve linguistic fidelity. As shown in Figure 1 , our method effectively prevents the model from defaulting to English in non-English queries. To the best of our knowledge, no previous work has achieved full multilingual competence in VLMs through such a simple and scalable approach. Our contributions are as follows: • We systematically demonstrate that integrating multilingual text-only data during training significantly reduces IFL bias in LLaVA-style VLMs while maintaining core capabilities. • We conduct an extensive analysis on the optimal proportion of text-only data required for effective multilingual adaptation. • We explore a model merging strategy, combining the visually fine-tuned model with the original multilingual backbone LLM, and assess its impact on preserving linguistic fidelity. Our findings suggest that we can develop high-quality multilingual VLMs that maintain strong performance across multiple languages in a simple and scalable way. By avoiding the need to translate or construct multimodal datasets for each language, our approach lowers the entry barrier for multilingual VLM development. This makes it especially attractive for low-resource settings, where monolingual text is often available but collecting vision-language data is costly or impractical. 2 Related Work 2.1 Multimodal Large Language Models VLMs typically integrate an image encoder, usually CLIP ( clip ; Dosovitskiy et al., 2021 ) , with an LLM backbone. Various strategies exist for combining these components. The predominant approach follows a decoder-only architecture, as seen in the LLaVA series, where an adapter module projects visual representations into the textual embedding space. Other methods include cross-attention mechanisms ( Grattafiori et al., 2024 ) , and some models, like NVLM Dai et al. (2024) , adopt a hybrid strategy combining both approaches. LLaVA-style models tend to default to English due to the scarcity of multimodal training data in other languages ( Hinck et al., 2024 ) . This issue arises because the LLM’s parameters are updated for a distinct task, which can disrupt its original language capabilities. Llama 3 ( Grattafiori et al., 2024 ) takes a different approach by freezing the LLM during training, which helps preserve its pretrained abilities while incorporating visual information. However, freezing the LLM also limits the model’s capacity to learn new visual tasks, creating a trade-off between language preservation and multimodal learning. 2.2 Multilingual Multimodal Learning A widely adopted approach to improving multilinguality in VLMs is translating existing multimodal datasets. Several works ( song2024missing ; Hu et al., 2024 ) have analyzed this strategy and proposed methods to enhance its effectiveness. Several models, such as PALI Chen et al. (2023) , PALI-X ( Chen et al., 2024d ) , mBLIP ( Geigle et al., 2024 ) , PALO ( Maaz et al., 2024 ) and Pangea ( pangea ) , have pursued this approach. However, this strategy presents challenges, including computational overhead, translation inconsistencies, and the loss of cultural context in visual-text pairs. Moreover, recent research ( Aggarwal et al., 2024 ) suggests that continual fine-tuning can harm an LLM’s performance. When a model undergoes two consecutive fine-tuning phases with differing task distributions, its ability to perform earlier tasks deteriorates. This raises concerns that direct fine-tuning solely on translated multimodal data may degrade the LLM’s original capabilities. 2.3 Catastrophic Forgetting Prevention In the context of LLMs, the problem of maintaining performance across tasks while integrating new information is known as lifelong learning. This field focuses on a system’s ability to acquire, integrate, and retain knowledge without catastrophically forgetting previous information. Visual Instruction Tuning is a case of lifelong learning, and it faces the same challenges. One known mitigation strategy is episodic or experience replay ( zheng2025survey ) , which helps prevent catastrophic forgetting by reintroducing previously learned information. Several studies ( Liu et al., 2022 ; Ibrahim et al., 2024 ) have explored ways to incorporate pretraining data during fine-tuning. Bethune et al. (2025) further analyze the impact of this approach and suggest that even a small amount of pretraining data can help retain previously learned knowledge, reducing the risk of performance degradation. In the case of VLMs, NVLM ( Dai et al., 2024 ) and InternVL 2.5 ( Chen et al., 2024e ) demonstrate that incorporating high-quality text-only data during Visual Instruction Tuning, not only improves the overall text-generation capabilities, but also multimodal performance. Our approach builds upon these findings by integrating multilingual text-only data throughout VLM training to mitigate IFL, without requiring extensive multimodal multilingual data collection. 2.4 Model Merging Model merging is a technique that involves combining two or more pre-trained models to create a new model that leverages the strengths of each. By merging a fine-tuned model with its original backbone, this process preserves the model’s prior capabilities while incorporating additional refinements from further training. This strategy has been applied in various contexts, such as language transfer, where Alexandrov et al. (2024) demonstrate that

Read the complete text