Metadata Conditioned Large Language Models for Localization

Anjishnu Mukherjee Ziwei Zhu Antonios Anastasopoulos · 2026

Read Metadata Conditioned Large Language Models for Localization by Anjishnu Mukherjee Ziwei Zhu Antonios Anastasopoulos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15236.txt.

Introduction

Anjishnu Mukherjee, Ziwei Zhu, and Antonios Anastasopoulos teach language models a sense of place. Their 2026 arXiv paper deals with a common flaw: models are trained on text as one global mix. The result is behavior that is geographically homogenized, the same everywhere.

The team studies metadata conditioning as a light fix. They pre-train dozens of small models from scratch on English news text. Each text is tagged with its source URL, country, and continent. The data spans four continents and seventeen countries, and the tags steer the models toward local norms.

Catalog record

CREATORAnjishnu Mukherjee Ziwei Zhu Antonios Anastasopoulos
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-15236

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Metadata Conditioned Large Language Models for Localization Anjishnu Mukherjee Ziwei Zhu Antonios Anastasopoulos Abstract Large language models are typically trained by treating text as a single global distribution, often resulting in geographically homogenized behavior. We study metadata conditioning as a lightweight approach for localization, pre-training 31 31 models (at 0.5 0.5 B and 1 1 B parameter scales) from scratch on large-scale English news data annotated with verified URLs, country tags, and continent tags, covering 4 4 continents and 17 17 countries. Across four controlled experiments, we show that metadata conditioning consistently improves in-region performance without sacrificing cross-region generalization, enables global models to recover localization comparable to region-specific models, and improves learning efficiency. Our ablation studies demonstrate that URL-level metadata alone captures much of the geographic signal, while balanced regional data coverage remains essential, as metadata cannot fully compensate for missing regions. Finally, we introduce a downstream benchmark of 800 800 localized news MCQs and show that after instruction tuning, metadata conditioned global models achieve accuracy comparable to LLaMA-3.2-1B-Instruct , despite being trained on substantially less data. Together, these results establish metadata conditioning as a practical and compute-efficient approach for localization of language models. 1 1 1 Code available here: https://github.com/iamshnoo/metadata_localization 1 Introduction Large Language Models (LLMs) are trained on large corpora drawn from the web, literary works, and news articles among other sources. Standard language modeling objectives treat all documents as a single global text distribution, often resulting in “western-aligned” behavior ( Weidinger et al. 2022 ; Ghosh et al. 2025 ) . However, knowledge is frequently localized and can vary substantially across regions. For instance, while a bride’s dress is typically white in Western weddings such as in the USA, it is often red in many Asian countries, including China, Japan, and India. When humans are asked an under-specified question ( Parrish et al. 2022 ) about the color of a wedding dress ( Yin et al. 2022 ) , they either answer based on their local context or request clarification. Both responses reflect a meta-understanding of localization that is often missing in western-centric LLMs. Figure 1: We pre-train LLMs with metadata augmented news articles, and evaluate them to determine the effect of metadata conditioning on localization. Existing approaches to localization are usually post hoc, relying on modified prompts at inference time ( Adilazuarda et al. 2024 ; Cao et al. 2025 ) or fine-tuning parameters to improve performance on localized benchmarks ( Choenni et al. 2024 ) . These methods treat localization as a control problem rather than a property learned during training, and often generalize unevenly across contexts. Other work advocates continued pre-training ( Gururangan et al. 2020 ) or modified objectives incorporating geographic signals such as latitude and longitude ( Hofmann et al. 2024 ) . While effective, these approaches are computationally expensive, limiting their practical applicability. We propose localizing modern LLMs by explicitly incorporating metadata that is already available in many training corpora (Figure 1 ). Specifically, we test the hypothesis that conditioning training documents on metadata improves localization without sacrificing global generalization. To this end, we pre-train LLMs on a large-scale news dataset annotated with verified URL, country, and continent tags, using a standard language modeling objective with metadata prepended to each document. \mathbf{(1)} We train both local models for individual continents and global models trained on data from all continents. \mathbf{(2)} We conduct controlled ablations to examine the effect of metadata granularity. \mathbf{(3)} We also evaluate leave-one-out global models that exclude a single continent to assess the importance of balanced regional data. \mathbf{(4)} Finally, we instruction-tune the models for chat-based inference and compare their downstream performance against LLaMA-3.2-1B-Instruct on our benchmark of localized knowledge. Our experiments use the LLaMA 3 architecture ( Grattafiori and Team 2024 ) and the News on the Web corpus ( Davies 2016 , Now ;) . All models are pre-trained with identical hyperparameters and observe the same total number of tokens, enabling fair controlled comparisons at evaluation time. We train two model sizes, 0.5 B and 1 B parameters, to study the effect of scale within our compute budget. We find that including metadata during pre-training and inference leads to better localized models, by designing four experiments: \bullet In Experiment \mathbf{1} , we compare metadata conditioned local models ( [Local] ) against non-conditioned controls ( Local ) across four continents, and find that metadata conditioning consistently reduces test perplexity. \bullet In Experiment \mathbf{2} , we compare metadata conditioned global models ( [Global] ) with local variants ( [Local] ) and find that they have similar test perplexities. \bullet In Experiment \mathbf{3} , we conduct ablations over metadata granularity and training data selection. We find that URL-only metadata is already effective, while excluding any single region during training uniformly degrades performance. \bullet In Experiment \mathbf{4} , we construct a downstream benchmark to evaluate localized factual knowledge. After instruction tuning on non-overlapping data, [Global] achieves accuracy comparable to LLaMA-3.2-1B-Instruct . # Continents Train Metadata Infer Metadata Model 1 ✓ ✓ [Local] 1 ✓ \times [Local] 1 \times ✓ Local 1 \times \times Local 4 ✓ ✓ [Global] 4 \times \times global Table 1: We adopt this notation for model variants. Models with bordered boxes are conditioned on metadata at inference time, while models whose names include […] are pre-trained with metadata conditioning. 2 Related Work Metadata conditioning has been studied as a mechanism for controlling language model behavior. Early work by Keskar et al. 2019 showed that control codes can steer stylistic and topical attributes. This idea was later extended to temporal settings, where Dhingra et al. 2022 demonstrated that conditioning on time enables models to act as updatable, time-aware knowledge bases. More recent work has examined the role of metadata in scaling knowledge capacity, with Allen-Zhu and Li 2025 showing that synthetically generated biographical metadata can improve memorization in LLMs. Metadata for conditioning can take many forms. In multilingual settings, Liu et al. 2020 use document language identifiers to enable effective multilingual denoising pre-training. Structural metadata has also been explored by Aghajanyan et al. 2022 who leverage HTML tags during pre-training and prompting to better model hypertext structure. Other work incorporates signals not directly observable in raw text, such as reward model scores used during pre-training to align models with human preferences ( Korbak et al. 2023 ) , or document identifiers to enable source-aware generation and knowledge attribution ( Khalifa et al. 2024 ) . Metadata cues are also effective during inference. Weller et al. 2024 show that appending phrases such as “according to …” helps models retrieve and quote from specific pre-training sources. Despite these advances, much of the prior work relies on smaller models, older architectures, synthetic metadata, or narrowly scoped tasks. A recent exception is Gao et al. 2025 , who study metadata conditioning in modern LLMs using real URLs and show that it accelerates learning during pre-training. However, their work does not examine how metadata used in pre-training interacts with post-training procedures. Our work builds on this line of research but addresses complementary questions. Like Gao et al. 2025 , we use modern architectures and real URLs as metadata; however, we additionally incorporate verified country and continent tags and focus explicitly on localization.

Read the complete text