On Generalization across Measurement Systems: LLMs Entail More Test-Time Compute for Underrepresented Cultures

Minh Duc Bui, Kyung Eun Park, Goran Glavaš, Fabian David Schmidt, Katharina von der Wense · 2025

Read On Generalization across Measurement Systems: LLMs Entail More Test-Time Compute for Underrepresented Cultures by Minh Duc Bui, Kyung Eun Park, Goran Glavaš, Fabian David Schmidt, Katharina von der Wense (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02591.txt.

Introduction

People switch between measurement systems with ease, since conversions are fixed, but language models may not. Minh Duc Bui and four co-authors test seven open models in 2025 on facts stated in different systems, from currencies to weights. They ask which system each model reaches for by default and how steady its answers stay across systems.

Models defaulted to the system that dominates their training data, the team found. Answers wobbled in both wording and accuracy when the system changed. Step by step reasoning could steady the answers, but at the price of far longer replies and higher running cost. The authors say that price falls hardest on users from cultures whose systems are rare in the data.

Catalog record

CREATORMinh Duc Bui, Kyung Eun Park, Goran Glavaš, Fabian David Schmidt, Katharina von der Wense
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02591

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Generalization across Measurement Systems: LLMs Entail More Test-Time Compute for Underrepresented Cultures Minh Duc Bui, Kyung Eun Park, Goran Glavaš, Fabian David Schmidt, Katharina von der Wense Abstract Measurement systems (e.g., currencies) differ across cultures, but the conversions between them are well defined so that humans can state facts using any measurement system of their choice. Being available to users from diverse cultural backgrounds, large language models (LLMs) should also be able to provide accurate information irrespective of the measurement system at hand. Using newly compiled datasets we test if this is the case for seven open-source LLMs, addressing three key research questions: (RQ1) What is the default system used by LLMs for each type of measurement? (RQ2) Do LLMs’ answers and their accuracy vary across different measurement systems? (RQ3) Can LLMs mitigate potential challenges w.r.t. underrepresented systems via reasoning? Our findings show that LLMs default to the measurement system predominantly used in the data. Additionally, we observe considerable instability and variance in performance across different measurement systems. While this instability can in part be mitigated by employing reasoning methods such as chain-of-thought (CoT), this implies longer responses and thereby significantly increases test-time compute (and inference costs), marginalizing users from cultural backgrounds that use underrepresented measurement systems. 1 Introduction Figure 1: We test whether LLMs can translate facts across different measurement systems and show that LLMs often struggle to provide accurate information for cultural contexts underrepresented in training data. The user base of large language models (LLMs) expands to people from a wide range of cultural backgrounds, with each person bringing distinct perspectives to their interactions with these ubiquitous systems. Intuitively, users expect that data can be correctly presented in ways that align with their cultural context, including measurement systems that they are used to. For instance, individuals accustomed to the metric system might find it challenging to interpret imperial units. Prior work shows that LLMs encode factual information, including plethora of measurement values, within their parameters ( Petroni et al. 2019 ; Geva et al. 2023 ; Allen-Zhu and Li 2024 , inter alia ) . However, while an LLM may accurately present facts within one cultural context (presumably the one that dominated their respective training corpora), an inclusive system should be able to do so in all contexts. By failing to do so, LLMs would disadvantage users from cultural backgrounds that are less represented in their training data. This raises a critical, yet underinvestigated question: can LLMs effectively generalize across socio-cultural contexts, offering same levels of factual correctness irrespective of the concrete culturally-bound factors such as measurement systems? Or do they exhibit biases that hinder socio-cultural adaptability, preventing them from equitably providing accurate information to all users? In this work, we demonstrate that LLMs are less reliable when the users’ measurement system differs from the dominant system of the LLM. We find that LLMs typically default to the measurement that fits the cultural context of the data, e.g., reporting fiscal values in USD as well as length and weight in the metric or imperial system, depending on the context of the data. For instance LLMs report food prices of US goods in USD normalized per pounds. The main issue arises when users naively prompt the models with systems that differ from the dominant one, as LLMs often fail to successfully translate values from one measurement into another (e.g., monetary amounts from USD to SDG). While advanced prompting techniques such as chain-of-thought reasoning largely mitigate these conversion errors (e.g., from 72% to almost no difference for currency for the largest LLMs in our evaluation), we need to note that this improvement comes at a substantial cost. Under CoT, models produce much longer, more verbose responses as they triangulate the conversion process step-by-step. For example, obtaining the correct value for the GDP of a country in Eritrean Nakfa can increase the inference cost by up to 300% 1 1 1 Our estimates are based on API pricing from https://groq.com/pricing/ for Llama 3.3 70B. compared to requesting the same value in US Dollars. Such issues have severe implications for users from marginalized cultural backgrounds, who are meant to benefit the most from equitable access to language technology. When LLMs misrepresent or inaccurately convert critical measurements (e.g., wrongly measuring monetary value of consumer goods), they alienate users by failing to align with their cultural context. Moreover, as these users frequently come from economically disadvantaged regions, they are disproportionately affected by the increased inference costs associated with extended model responses. We address the following research questions in this work (together with respective contributions). (RQ1) What measurement do LLMs default to? We curate datasets that span diverse cultural contexts, including fiscal data, food prices, and city distances across different countries and regions. We find that LLMs default to the measurement that fits the cultural context of the data. When queried on distances between German and US cities, they provide estimates in the metric and imperial units, respectively. Likewise, LLMs express fiscal data in USD, the common convention for financial data. (RQ2) Do LLMs’ answers and their accuracy vary across measurement systems? We test whether varying measurement systems affects factual accuracy of answers provided by LLMs (as illustrated in Figure 1 ). We find that LLMs exhibit significant variability in behavior depending on the measurement system embedded in the prompt. Models perform best when the query aligns with the measurement system that fits the cultural context of the data (e.g., provided by a location or entity mentions) . For instance, models show a strong decline in accuracy when asked about food prices in pounds for a country that uses kilograms. (RQ3) Can LLMs mitigate these challenges with reasoning? We find that LLMs often struggle to generate accurate facts across different measurement systems. Nevertheless, very large LLMs (70B parameters) can successfully perform sequential single-hop inferences, starting with the default measurement and then converting the value to the target unit. This exemplifies that LLMs necessitate additional guidance to reliably triangulate to the correct value in the target measurement. Explicitly prompting LLMs to reason across measurement systems similarly stabilizes performance. Yet, such reasoning remains anchored in the default measurement, increasing inference costs for queries reflecting non-default cultural contexts. Fiscal Data City Distances Food Prices per Weight Measurement System Currency Length Systems Weight Systems Units USD, EUR, VND, Kilometer (Metric), Kilogram (Metric), KRW, CNY, JPY, … Mile (Imperial), Pound (Imperial) Cape Foot (South African), Jin (Chinese), Wa (Thai), Geun (Korean), Li (Chinese), Catty/Kati (Malaysian), … … Number of Systems 112 10 10 Total Samples 16576 31200 (Kilometer), 24130 (Kilogram), (Facts \times Systems) 7800 (Mile) 3690 (Pound) Number of Facts 148 3120 (Kilometer), 780 (Mile) 2413 (Kilogram), 369 (Pound) Example Question What is GDP per capita for What is the distance between What is the price of one Korean Australia in 2021 in JPY ? New York and Los Angeles in Geun of steak (beef) in Nigeria, Li (Chinese) ? in Nigerian Naira (NGN)? Table 1: Measurement System Datasets: Measurement system and units, as well as sample size by dataset type. For further information about each measurement system, see Appendix Table 5 . 2 Measurement Systems & Datasets 2.1 Measurement Systems Each dataset is rooted in a specific cultural

Read the complete text