Understanding Inequality of LLM Fact-Checking over Geographic Regions with Agent and Retrieval models
Bruno Coelho, Shujaat Mirza, Yuyuan Cui, Christina Pöpper, Damon McCoy · 2025
Read Understanding Inequality of LLM Fact-Checking over Geographic Regions with Agent and Retrieval models by Bruno Coelho, Shujaat Mirza, Yuyuan Cui, Christina Pöpper, Damon McCoy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22877.txt.
Introduction
Bruno Coelho and co-authors test in 2025 how well large language models check facts from six world regions. False claims spread fast, and checking them by machine could help. Yet model skill may vary by place, which would leave some regions worse served.
The team used six hundred checked claims split evenly across regions and tried three setups. One gave the model the claim alone, one added web-like search of an encyclopedia, and one gave the true check itself. In every setup, claims from the Global North scored far better than claims from the Global South. The gap grew with the search setup, showing broad sources miss local detail.
Catalog record
| CREATOR | Bruno Coelho, Shujaat Mirza, Yuyuan Cui, Christina Pöpper, Damon McCoy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22877
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding Inequality of LLM Fact-Checking over Geographic Regions with Agent and Retrieval models Bruno Coelho, Shujaat Mirza, Yuyuan Cui, Christina Pöpper, Damon McCoy Abstract Fact-checking is a potentially useful application of Large Language Models (LLMs) to combat the growing dissemination of disinformation. However, the performance of LLMs varies across geographic regions. In this paper, we evaluate the factual accuracy of open and private models across a diverse set of regions and scenarios. Using a dataset containing 600 fact-checked statements balanced across six global regions we examine three experimental setups of fact-checking a statement: (1) when just the statement is available, (2) when an LLM-based agent with Wikipedia access is utilized, and (3) as a best case scenario when a Retrieval-Augmented Generation (RAG) system provided with the official fact check is employed. Our findings reveal that regardless of the scenario and LLM used, including GPT-4, Claude Sonnet, and LLaMA, statements from the Global North perform substantially better than those from the Global South. Furthermore, this gap is broadened for the more realistic case of a Wikipedia agent-based system, highlighting that overly general knowledge bases have a limited ability to address region-specific nuances. These results underscore the urgent need for better dataset balancing and robust retrieval strategies to enhance LLM fact-checking capabilities, particularly in geographically diverse contexts. Keywords fact-checking \cdot regional diversity \cdot agent-based systems 1 Introduction The digital age has amplified our access to information, but in the same stride, it has magnified the challenge of mis-/disinformation [ Caramancion(2020) ] . The resulting wave of misleading content risks eroding the very foundations of our informed societies. Now, more than ever, as societies navigate through an intricate web of data-driven decisions, the integrity and trustworthiness of our information sources are under scrutiny [ Shu et al.(2020) , Grimes(2021) , Bovet and Makse(2019) ] . Large Language Models (LLMs) have experienced widespread adoption due to their advanced capabilities in processing complex information [ Paul et al.(2023) ] . Recognizing their potential impact, there has been an increasing focus on aligning these models with facts through Reinforcement Learning from Human Feedback (RLHF) to avoid harmful content generations [ Ouyang et al.(2022) ] . As users increasingly place trust in LLMs with the responsibility of discerning fact from fiction [ Lee et al.(2021) , Guo et al.(2022) , Wahle et al.(2022) , Aggarwal et al.(2020) , Pelrine et al.(2021) ] , ensuring their factuality and fairness becomes a central concern. The impact of mis-/disinformation varies globally, influenced by factors like regional digital literacy and the strength of local fact-checking ecosystems. Regions with lower digital literacy are often more susceptible to misinformation due to the lack of robust mechanisms to critically assess and verify digital content [ Mirza et al.(2023) ] . Such disparities highlight the need for examining the performance of LLMs across different regions. If these models are not finely attuned to regional variations, they risk exacerbating existing informational inequities. Our study, therefore, delves into the geographic factuality variations of LLMs, addressing an important aspect of how these technologies can either mitigate or amplify challenges in diverse information ecosystems. It is crucial to understand these disparities as they can disproportionately affect underrepresented regions where reliable fact-checking resources may be scarce. To evaluate these issues, we employ a balanced dataset comprising 600 fact-checked statements distributed evenly across six global regions. We observe notable performance disparities in LLMs based on geographic origin, uncovering a concerning bias where all of the LLMs that we studied more accurately fact-checked statements from the Global North (e.g., North America) than those from the Global South (e.g. Africa). Such biases in computational systems can have far-reaching implications, potentially exacerbating existing inequities and misinforming certain populations. Next, we explore methods to improve the accuracy of LLMs when fact-checking statements. Ideally, we sought to discover methods that provide improvements and parity when fact-checking statements from different regions of the world. Our methods focus on providing the LLMs with access to external knowledge bases that might be useful for fact-checking. We explore multiple configurations, including the baseline statement-only setups, retrieval-augmented generation (RAG), and agent-based fact-checking scenarios. By examining these configurations, we aim to understand the strengths and limitations of LLMs under varying degrees of access to external information. The different scenarios reveal a complex landscape. LLMs demonstrate high accuracy (exceeding 98 %) when provided with the correct document, highlighting their potential as effective tools in controlled environments. Unfortunately, their performance declines significantly in agent-based scenarios where models interact with a Wikipedia external knowledge-base. These declines are further pronounced for statements originating from the Global South, adding to the already existing gap between the regions. Such results emphasize the urgent need for geographically balanced datasets and model training to address these inequities. Our major contributions include: 1. We highlight significant geographic disparities in fact-checking accuracy, with statements from the Global South consistently under-performing compared to those from the Global North. By quantitatively uncovering these discrepancies, our research highlights the need for culturally aware and geographically diverse model training and evaluation, which is vital for achieving equitable computational systems worldwide. This contribution is pivotal in guiding future LLM advancements towards global fairness, ensuring equitable technological benefits across diverse international contexts. 2. We explore whether adding external knowledge bases improves accuracy and reduces inequality across different fact-checking scenarios, including statement-only, RAG, and agent-based setups, offering nuanced insights into the strengths and limitations of current LLMs in the presence of external knowledge bases. 2 Related Work The recent advances in LLMs have been studied for many NLP tasks such as text summarization [ Yang et al.(2023) , Wei et al.(2023) ] , entity recognition [ Omar et al.(2023) , Hu et al.(2023) ] , and question and answering [ Katz et al.(2023) , OpenAI(2023) , Huh(2023) , Qin et al.(2023) ] . While impressive, these models are not immune to limitations and may raise their own problems. OpenAI has warned, “GPT-4 ‘hallucinates’ facts and makes reasoning errors” to some extent [ OpenAI(2023) ] , Anthropic claims its Claude-series of models “can occasionally produce responses that are incorrect or misleading” [ Anthropic(2024a) ] , and Meta warns users on the LLaMA 3.3 model page that it “may in some instances produce inaccurate, biased or other objectionable responses to user prompts” [ Meta(2024) ] . Alongside the advancements in NLP, there is a growing concern over the impact of digital misinformation [ Lewandowsky et al.(2017) ] . The last few years have seen its proliferation into subjects such as climate change [ Biddlestone et al.(2022) , van der Linden et al.(2017) ] , vaccinations, and COVID-19 [ Vidgen et al.(2021) , He et al.(2023) ] . The capabilities of LLMs raise important questions about their role as both a mitigator and generator of misinformation, either at the user level or as part of a larger system. 2.1 Fact-Checking Capabilities of LLMs. Most of the work looking at freely available products has focused on ChatGPT and textual fact-checking, although newer multimodal models provide the chance to also fact-check images or videos [ Geng et al.(2024) ] . ChatGPT has been used for examining both vaccination and cancer misconceptions [ Deiana et al.(2023) , Johnson et al.(2023) ] , both finding it