Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries

Damian Curran, Vanessa Sporne, Dr. Lea Frermann, Prof. Jeannie Paterson · 2025

Read Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries by Damian Curran, Vanessa Sporne, Dr. Lea Frermann, Prof. Jeannie Paterson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06700.txt.

Introduction

Damian Curran, Vanessa Sporne, Lea Frermann, and Jeannie Paterson test whether chatbots give equally reliable legal information in different cities. Comparing legal knowledge across places is tricky because institutions differ, so the team borrowed functionalism from comparative law, matching scenarios rather than statutes. Real advice-seeking posts from Reddit supplied family, housing, work, crime, and traffic situations.

Each scenario was posed for Los Angeles, London, and Sydney, and the legal summaries returned were hand-checked for invented content. Hallucination rates differed significantly by place among leading closed models, so answer quality is unevenly spread across geography. A model's self-consistency across repeated samples tracked its error rate closely, offering a practical uncertainty signal.

Catalog record

CREATORDamian Curran, Vanessa Sporne, Dr. Lea Frermann, Prof. Jeannie Paterson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06700

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries Damian Curran, Vanessa Sporne, Dr. Lea Frermann, Prof. Jeannie Paterson Abstract How do we make a meaningful comparison of a large language model’s knowledge of the law in one place compared to another? Quantifying these differences is critical to understanding if the quality of the legal information obtained by users of LLM-based chatbots varies depending on their location. However, obtaining meaningful comparative metrics is challenging because legal institutions in different places are not themselves easily comparable. In this work we propose a methodology to obtain place-to-place metrics based on the comparative law concept of functionalism. We construct a dataset of factual scenarios drawn from Reddit posts by users seeking legal advice for family, housing, employment, crime and traffic issues. We use these to elicit a summary of a law from the LLM relevant to each scenario in Los Angeles, London and Sydney. These summaries, typically of a legislative provision, are manually evaluated for hallucinations. We show that the rate of hallucination of legal information by leading closed-source LLMs is significantly associated with place. This suggests that the quality of legal solutions provided by these models is not evenly distributed across geography. Additionally, we show a strong negative correlation between hallucination rate and the frequency of the majority response when the LLM is sampled multiple times, suggesting a measure of uncertainty of model predictions of legal facts. [orcid=0009-0006-1440-311X, [email protected], ] *1 [orcid=0000-0002-9712-1188, [email protected], ] [orcid=0000-0002-2649-7591, [email protected], ] keywords Access to Justice ,Natural Language Processing ,Large Language Models ,Hallucination ,Geographic Bias ,Comparative Law ,Automated Hallucination Detection † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: AI4A2J-ICAIL25: AI for Access to Justice at the International Conference on AI and Law 2025, June 20, 2025, Chicago, IL † † address: School of Computing and Information Systems, The University of Melbourne, Australia † † address: The Centre for Artificial Intelligence and Digital Ethics † † address: Melbourne Law School, The University of Melbourne, Australia † † corresp: Corresponding author. 1 Introduction How do we make a meaningful comparison of a large language model’s ( LLM ) knowledge of the law in one place compared to another? For instance, does an LLM have ’better’ knowledge of the laws of Los Angeles, USA, than the laws of London, UK? Will a person using an LLM chatbot to assist with a legal problem in Sydney, Australia, be provided with legal information that is equally as accurate as a person in a similar situation, but subject to the laws of Toronto, Canada? Or Singapore? Or Auckland? These questions are important because a growing body of survey work suggests that ordinary people across the globe (including Czechia Kuk and Harasta 2025 , the UK Seabrooke et al. 2024 and California Hagan 2024 ) are increasingly using LLM-based chatbots to obtain legal information for personal matters. The ostensible reason for the suitability of LLM chatbots for these tasks is obvious - LLM access is either free or relatively low cost, has an accessible chat interface and can provide personalized information, targeted to individual situations. But, as noted by Dahl et al., the potential for LLMs to close the access to justice gap will only be reached if LLMs ’actually know the law’ Dahl et al. 2024 . If an LLM provides inaccurate legal information which nonetheless sounds plausible, it may in fact have the opposite effect on the justice gap by systematically disadvantaging those who need the most help. In this work we explore whether the accuracy of legal information produced by LLMs varies for different users across geographic boundaries. It is a trite observation - but one worth noting explicitly - that laws vary from place to place. Whilst some high-level norms may be globally consistent - such as the near universal prohibition on violent crime - the actual form and manifestation of those norms will vary, often to a local province or county level. Very little of the legal landscape is truly international and those few subject areas that are international have very little bearing on the typical pro se (or ’self-represented’) litigant seeking legal advice for a personal matter, such as a family, employment or tenancy dispute. Most such laws vary along national and often local boundaries and so an LLM’s knowledge of such a law in one place will not necesarily be indicative of comparable knowledge in another. Understanding how hallucination rates vary across geographic bounds is therefore critical for evaluating whether access to justice can be delivered equitably to all LLM users, regardless of where they live. In this work we propose a new methodology for obtaining a metric of LLM hallucination rates which can be used to meaningfully compare LLM legal knowledge across geographic lines. We discuss the method’s theory, drawing on the comparative law concept of functionalism. We provide empirical evidence that the rate of hallucination of legal facts by leading closed-source LLMs is significantly associated with place and we begin a discussion on possible correlating factors based on observations in our data. Figure 1: Overall workflow : A subject matter-diverse set of Reddit posts were converted to concise, place-agnostic scenerios. The prompt template has slots for the ’place’ and the converted ’scenario’. The source text of the pinpoint reference identified by the LLM is found online, and the LLM’s summary is manually annotated for hallucinations by a human annotator. 2 A Comparative Law Approach Producing meaningful comparative metrics of an LLM’s knowledge of laws in different places is challenging. One method that might initially be considered is a comparison of the hallucination rates on datasets of conceptually similar legal instruments in different places, such as the body of laws passed by the legislatures in the places being examined. But whilst these metrics would be useful for evaluating and improving the localized performance of the models in each place, there are significant challenges in using them as the basis for an ’apples-to-apples’ comparison of LLM knowledge across places. This is because each body of legislation has a different size and subject matter. They also operate in different historical, political and cultural contexts. Thankfully, this is not a new conceptual problem. Comparative lawyers have thinking about how to compare different legal systems for generations. One method in particular has come to dominate modern comparative law studies – namely, functionalism. Functionalism can be defined as having at its core ’the comparison of solutions which different legal orders offer for specific practical problems’ Kischel 2019 . Instead of attempting to compare conceptually similar legal institutions and instruments, we instead begin with specific practical problems, and compare the solutions offered to those problems in each place. We adopt a functionalist approach in this study by framing our dataset as a series of ’practical problems’ to which the LLM needs to identify a solution in each place. The ’solution’, typically a reference to a legal instrument applicable in the place, is then evaluated for hallucinations. The resultant hallucination rates can then be meaningfully compared, because they reflect the LLMs relative knowledge of the solutions to the same practical problems.

Read the complete text