Language Bias under Conflicting Information in Multilingual LLMs
Robert Östling, Murathan Kurfalı · 2026
Read Language Bias under Conflicting Information in Multilingual LLMs by Robert Östling, Murathan Kurfalı (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07123.txt.
Introduction
Östling and Kurfalı test whether multilingual language models favour one tongue over another when facts clash. Models are known to fuse clashing facts with bias, but the authors ask if the tongue carrying each fact sways the choice. They stretch the conflicting needles test to five tongues with natural news data, across many multilingual models of varied size. All tested models mostly ignore the clash and firmly state just one answer, with a steady bias against Russian and, at the longest contexts, in favour of Chinese.
Catalog record
| CREATOR | Robert Östling, Murathan Kurfalı |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50294 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.07123
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language Bias under Conflicting Information in Multilingual LLMs Robert Östling, Murathan Kurfalı Abstract Large Language Models (LLMs) have been shown to contain biases in the process of integrating conflicting information when answering questions. Here we ask whether such biases also exist with respect to which language is used for each conflicting piece of information. To answer this question, we extend the conflicting needles in a haystack paradigm to a multilingual setting and perform a comprehensive set of evaluations with naturalistic news domain data in five different languages, for a range of multilingual LLMs of different sizes. We find that all LLMs tested, including GPT-5.2, ignore the conflict and confidently assert only one of the possible answers in the large majority of cases. Furthermore, there is a consistent bias across models in which languages are preferred, with a general bias against Russian and, for the longest context lengths, in favor of Chinese. Both of these patterns are consistent between models trained inside and outside of mainland China, though somewhat stronger in the former category. 1 Introduction Given the enormous impact and real-world deployment of large language models (LLMs) in recent years, it is essential to evaluate how these models deal with the complexity of information that they are used to process. One particular such complexity is contradictory information, which is abundant in today’s information landscape. A very common application for LLMs is to summarize information from multiple sources, and if sources disagree on some points, it is of critical importance to know how these disagreements are reflected in the output. Previous work ( 6 ) has investigated this question with respect to factors such as repetition and positioning of information in the context window given to models, by using a conflicting needles in a haystack paradigm where conflicting pieces of information ( needles ) are inserted into large amounts of unrelated textual information ( haystacks ). We expand on this line of research by systematically varying the languages of the needles and haystacks in order to test whether the language information is presented in affects its chances of making it to the final output. Specifically, we ask the following research questions: • RQ1: How do current multilingual LLMs behave when faced with conflicting information in a naturalistic information retrieval setting? • RQ2: Are current multilingual LLMs biased with respect to which language they prioritize when extracting contradictory information? • RQ3: Are any languages consistently favored or disfavored across LLMs? • RQ4: Does the country of origin of the LLMs influence their handling of contradictory information? We identify four major contributions of this paper, both methodological (1) and empirical on the (in)ability to deal with conflicting information (2) as well as a detailed look at how the biases made apparent by this failure are structured (3, 4). 1. We introduce the multilingual conflicting needles in a haystack paradigm for investigating language biases in multilingual LLMs, using manually constructed naturalistic templates that simulate a real-world information retrieval scenario. 2. Contrary to our expectation, we find that even the state-of-the-art models such as consistently fail to detect conflicting information, even under the easiest conditions in our experiments. 3. We find that there is a clear language bias as to which of two contradictory pieces of information is chosen by a model, and that this bias is surprisingly consistent across models. 4. We find that the patterns of bias are similar between models trained in the West (North America and Europe) as compared to the East (PRC), but that there seems to be a slight bias against English in PRC-trained models only. 2 Related Work The Needle In A Haystack (NIAH) paradigm, originally introduced by 5 , has been used in LLM research to test information retrieval under controlled circumstances. The key idea is to build a collection of text (“haystack”) into which one or more pieces of information (“needles”) are inserted. An LLM is then asked to retrieve the information, and the rate of success indicates how well the LLM under test is able to deal with this particular configuration of haystack and needles. 4 compared retrieval performance with contexts of different lengths, with the goal of mapping how usable context size compares to the architectural context size limit for common LLMs. Given their finding that many models show a substantial drop in retrieval performance with longer contexts, we systematically vary context length in our experiments. 13 extended the NIAH paradigm to multimodal data, using combined textual (English) and visual haystacks, finding that the accuracy for retrieving, counting, and reasoning from visual information was considerably lower than from information in text. 6 extended the NIAH paradigm to conflicting needles, where contradictory information is inserted into a haystack of text. They found that LLMs in general do not identify the conflict, and that in choosing which needle to retrieve, there are both model-specific biases and biases that are consistent across models. In particular, models tend to rely more on repeated needles. 9 extended this research to study how LLMs take source credibility into account when faced with conflicting information. They find that repetition of information can even make LLMs prioritize information from unknown or unreliable sources over high-credibility sources. While they were unable to mitigate this repetition bias with prompting-based approaches alone, they propose a fine-tuning method which reduces repetition bias and increases reliance on source credibility. Other mitigation approaches include training models to explicitly report sources for their claims ( 10 ) . 8 also studied how LLMs deal with conflicting information, both in a conflicting NIAH scenario with inserted needles, and by identifying naturally occurring contradictions within documents. They found that top models can identify about 60% of contradictory needles and achieve a precision of about 50% when identifying naturally occurring contradictions. Similar results have been found in other studies focusing on direct detection of contradictions ( 7 ; 12 ) . In general, this setting leads to the identification of a larger share of contradictions than the other studies referred to above ( 6 ; 9 ) , but their focus is on using state-of-the-art models with chain-of-thought (CoT), and prompting with the explicit goal of looking for contradictions. This contrasts with our focus, which is to test how contradictions are handled during “mundane” tasks like information retrieval. Given the different political systems, level of information control, and general cultural differences between mainland China (PRC) and what could be loosely referred to as the “west” (represented in our work by the US, Canada, and EU), it is reasonable to ask to what extent LLMs from these two areas are different. Previous work has shown that, for instance, the dominating political ideology in a country has an effect on the sentiments expressed by LLMs about prominent people ( 1 ) . We are interested in whether there are differences with respect to how different languages are treated. 14 found that in spite of what one may expect given the political and cultural differences, performance of PRC-trained LLMs is mostly in line with that of “western” models on a broad sample of languages including minority languages of the PRC, Mandarin, and several major languages from Asia and Europe. We test (RQ4) whether this result