Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent

2025

Read Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12838.txt.

Introduction

A model that answers in many languages should give the same facts in each of them, but that does not always happen. Xi Ai, Mahardika Krisna Ihsani, and Min-Yen Kan studied that cross-language match in 2025. They used mixed-language statements that state the same fact in two tongues and refer to the same things.

Tests on many base and tuned models showed uneven match across language pairs. Language family, writing script, and one layer deep in the model all played a part. Training with code-switching and word alignment helped most. A fix applied at test time, called activation patching, also showed promise for lifting the match.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12838

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent? Thanks: Equal contribution. Affiliation: Xi Ai, Mahardika Krisna Ihsani 1 1 footnotemark: 1 , Min-Yen Kan Affiliation: [1ex] School of Computing, National University of Singapore Thanks: Works conducted during the internship at NUS. Affiliation: Department of Natural Language Processing, MBZUAI [email protected] , [email protected] , [email protected] Abstract Cross-lingual consistency should be considered to assess cross-lingual transferability, maintain the factuality of the model knowledge across languages, and preserve the parity of language model performance. We are thus interested in analyzing, evaluating, and interpreting cross-lingual consistency for factual knowledge. We examine code-mixed coreferential statements conveyed identical knowledge across languages to study cross-lingual knowledge consistency. We use some interpretability approaches to analyze the behavior of a model in cross-lingual contexts, discovering that multilingual models show different levels of consistency, subject to language families, linguistic factors, and a bottleneck in cross-lingual consistency on a particular layer. In addition, we evaluate common strategies aimed at improving multilingual performance to observe whether these strategies can improve knowledge consistency at the same time. While knowledge is not cross-lingual consistency in many cases, code-switching training and cross-lingual word alignment objectives show the most promising results, emphasizing the noteworthiness of cross-lingual alignment supervision and code-switching training for both multilingual performance and cross-lingual consistency enhancement. \setcode utf8 1 Introduction Figure 1: Illustration of Cross-lingual knowledge consistency. Frege’s theory of reference defines the reference of a sub-sentential expression as the object singled out by the name. A salient aspect of humanity is that, they can understand knowledge based on references regardless of languages. Frege’s theory of reference ( Frege 1892 ) indicates that the knowledge conveyed by a sentence depends on the references of the expressions that make up the sentence. A salient aspect of humanity is that, while people may speak different languages, they can share common references and knowledge. Thus, references and knowledge must be consistent across languages, and a multilingual model serving as a knowledge base ( Gupta and Srikumar 2021 ; Kassner et al. 2021 ; Hu et al. 2024 ) should provide consistent knowledge when consulted in different languages. Not only does this theory contribute to cross-lingual performance and maintain knowledge between languages, but it ensures parity and self-consistency of model performance ( Hupkes et al. 2023 ; Wang et al. 2023 ) . This motivates us to evaluate the knowledge consistency of multilingual language models in all languages when sharing the same references. Although recent advances have shown that multilingual models are effective for cross-lingual transfer and generalization ( Conneau et al. 2020 ; Xue et al. 2020 ; Hu et al. 2020 ; Muennighoff et al. 2023 ) , Kassner et al. 2021 ; Fierro and Søgaard 2022 ; Qi et al. 2023a reported that prediction varies from one language to another when recalling knowledge in different languages. For given parallel statements, a language model may output predictions for a particular query that differ from the one obtained from the query’s translation. This examination implicitly instantiates Frege’s theory of reference to check the consistency of knowledge across languages that parallel statements with the same references for sub-sentential expressions, e.g. entities, should have the same knowledge. Inspired by that theory, we hypothesize that multilingual language models recall consistent factual knowledge for coreferential statements in cross-lingual settings. To evaluate this hypothesis (Figure 1 ), our idea is that recalling knowledge relies on the references of statements regardless of languages. Therefore, we examine code-mixed coreferential statements and attempt to answer two questions: 1) do multilingual language models recall factual knowledge for the coreferential statements in a similar manner? and 2) how does the mechanism of multilingual language models work on the incorporation between entities or references to convey knowledge in cross-lingual settings? Our study is related to a broader linguistic phenomenon of entity-level code-switching: an entity code-switches between two languages without changing the reference, as we create code-mixed coreferential statements from monolingual statements by substituting a subject entity with an equivalent one in another language that shares the same reference. More recently, we share a similar goal with knowledge incorporation and editing ( Beniwal et al. 2024 ; Li et al. 2024 ) , since we incorporate a coreferential entity from other languages to recall factual knowledge in cross-lingual settings. Our main findings are as follows. • We discover that scaling is not a promising strategy to improve cross-lingual consistency, as we identify consistency bottlenecks and issues tied to language characteristics and biases through layer-wise analyses and interpretability approaches. • There is a partial causality between adding monolingual biases and improving cross-lingual knowledge consistency. Directly adding bias via representation patching or expanding the vocabulary size could be a potential method to calibrate consistency. • Improving cross-lingual representation and multilingual performance might contribute to the cross-lingual consistency but not a sufficient method to achieve the cross-lingual consistency. Alignment inductive bias can alleviate the consistency bottleneck to enhance alignments between coreferential entities, which can be achieved by adding word alignment objective and code-switching training. However, multilingual instruction tuning cannot alleviate consistency bottlenecks, although it is a promising strategy to improve multilingual performance. Our contribution is to offer an understanding of multilingual language models’ limitations under cross-lingual settings and highlight potential research directions to address such issues. 2 Methodology 2.1 Task Definition We focus on a code-mixed context-independent cloze task that forces the multilingual model to rely on its internal knowledge base to recall the common knowledge shared by coreferential entities across languages due to cross-lingual generalization 1 1 1 See limitation in § 8 . . The high-level idea of this evaluation task is illustrated in Figure 1 where en entry "Paris is the capital of ___" is evaluated with its possible code-mixed coreferential statements (ar entry & ta entry). Readers can refer to Appendix § A.1 and § A.2 for details and implementations. Let I=\{S^{l1},\cdots,O,\cdots\}\in{l1} 2 2 2 The surface structure is not restricted. We use the common subject–object structure as an example. be a statement, where l1 stands for matrix language (the predominant language), S^{l1}=\{s_{1},\cdots,s_{k}\}\in l1 are subject sub-tokens, and O=\{o_{1},o_{2},\cdots,o_{j}\}\in l1 denote object sub-tokens. This statement is used to create a cloze task input I_{mono} for O with the n-gram prediction, denoted as Cand(O_{\in V}|I_{mono}) , as the top-k n-gram candidates obtained from beam search decoding over the model’s vocabulary V . Similarly, we can create a code-mixed coreferential statement I_{cm} by replacing S^{l1} with a coreferential subject S^{l2} in the embedded language l2 (the subsidiary language) in order to obtain Cand(O_{\in V}|I_{cm}) . Therefore, I_{cm} and I_{mono} are coreferential and expected to recall the same knowledge. We define the task as measuring the difference between output distributions, Cand(O_{\in V}|I_{cm})=P(O_{\in V}|K_{\theta}) P(K_{\theta}|S^{l2},I_{\setminus(S^{l1}\cap O)}) and Cand(O_{\in V}|I_{mono})=P(O_{\in V}|K^{*}_{\theta}) P(K^{*}_{\theta}|S^{l1},I_{\setminus(S^{l1}\cap O)}) , where K_{\theta} is the knowledge recalled from the model given the preceding context, and I_{\setminus(S^{l1}\cap O)} stands for I without both the subject and the object. Then, cross-lingual knowledge consistency between K^{*}_{\theta} and K_{\theta} reflects on the measured difference. Additionally, we also evaluated the baseline setting of I_{cm} by removing the subject entities to obtain the model’s default object tokens for comparison. 2.2 Metric Function and Interpretability Readers can refer to Appendix § A.3 for more details,

Read the complete text