Form and Meaning in Intrinsic Multilingual Evaluations
Wessel Poelman, Miryam de Lhoneux · 2026
Read Form and Meaning in Intrinsic Multilingual Evaluations by Wessel Poelman, Miryam de Lhoneux (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10580.txt.
Introduction
Wessel Poelman and Miryam de Lhoneux study how we judge language models that work in more than one language. Scores such as perplexity and bits-per-character are easy to use for one language. The authors argue that these scores rest on hidden assumptions once many languages are involved. Their 2026 arXiv preprint looks at what these measures really capture.
The paper asks what form and meaning each score reflects when text is compared across languages. A score that looks fair in one language may treat another language in a skewed way. The authors work through those assumptions one by one. Readers who build or test multilingual models get a clear warning about what the common scores can and cannot show.
Catalog record
| CREATOR | Wessel Poelman, Miryam de Lhoneux |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-10580
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Form and Meaning in Intrinsic Multilingual Evaluations Wessel Poelman, Miryam de Lhoneux Abstract Intrinsic evaluation metrics for conditional language models, such as perplexity or bits-per-character, are widely used in both mono- and multilingual settings. These metrics are rather straightforward to use and compare in monolingual setups, but rest on a number of assumptions in multilingual setups. One such assumption is that comparing the perplexity of CLMs on parallel sentences is indicative of their quality since the information content (here understood as the semantic meaning ) is the same. However, the metrics are inherently measuring information content in the information-theoretic sense. Consistency in meaning does not neutralize different forms (paraphrases). We make such assumptions explicit and discuss their implications. We perform experiments with six metrics on two multi-parallel corpora both with mono- and multilingual models. We find that current metrics are not universally comparable and look at the form-meaning debate to provide some explanation for this. 1 Introduction Intrinsic evaluation metrics, such as perplexity ( Jelinek et al. 1977 , PPL; ) , are often the first step in evaluating conditional language models (CLMs). PPL is the most ubiquitous, but other transformations of the loss (or negative log-likelihood) are also used. Intrinsic evaluations are sometimes the only option for certain languages due to resource availability Joshi et al. 2020 . There are two multilingual setups where these metrics are used: a single multilingual model or multiple monolingual models . In the multilingual model setting the intrinsic metrics indicate how well a single model learns multiple languages. Comparing metrics is used for model design: model A is better than B since A achieves a lower PPL . For the setting with multiple monolingual models, the metrics are used to study the interaction of language characteristics and language modeling: which language is hard to model? The comparisons are used to describe languages: the PPL for language A is lower than B, A is therefore easier to model . NLL ✓ NLL \times BPC ✓ BPC \times \text{DE}_{1}-\text{DE}_{4} EN \times / ✓ (In)consistent Figure 1: Each line represents a row in a parallel dataset, each dot is an individual sentence; four German and one English. Common multilingual evaluations center around comparing intrinsic metrics such as the negative log-likelihood (NLL) or bits per character (BPC) with the assumption that these comparisons are fair since the semantic meaning is consistent. However, these metrics measure information content in the information-theoretic sense. This results in (1) differences within a language ( \text{DE}_{i}\leftrightarrow\text{DE}_{k} ), and (2) inconsistency across languages: if the EN sentence falls outside the range of the DE sentences it is consistent . If it falls within the range it is inconsistent , meaning conclusions can flip depending on the DE sentence we choose in our test set. A common approach in both setups is to calculate the loss (or transformations of it) on multi-parallel datasets ( Cotterell et al. 2018 ; Gerz et al. 2018 ; Mielke et al. 2019 ; Park et al. 2021 ; Wan 2022 ; Chang et al. 2024a ; Chang et al. 2024b ; Limisiewicz et al. 2024 ; Arnett and Bergen 2025 , e.g., ) . A popular choice is the FLORES-200 machine translation dataset NLLB Team et al. 2022 . Multi-parallel datasets are presumed to allow for a fair evaluation: samples have the same semantic meaning in all languages. However, the fairness of comparing metrics using multi-parallel datasets is less clear. CLMs are optimized using an information-theoretic objective, namely to minimize the cross-entropy loss, which is explicitly not about semantics. We make such assumptions explicit and ask: 1. How does the distinction between semantic meaning and information content apply to intrinsic evaluation metrics? 2. What are the factors at play when comparing metrics, within and across languages? 3. What are assumptions about comparing metrics when using a single multilingual model or multiple monolingual models? To answer these questions, we first discuss six existing intrinsic metrics and show how they relate to each other. Next, we formalize the problem of comparing intrinsic metrics and relate it to the form-meaning debate. Third, we look at the behavior of the metrics on two parallel corpora for mono- and multilingual models. Fourth, we perform experiments with paraphrases to test the consistency of the metrics (Figure 1 ). Finally, we take a broader view and tie our findings to the larger question of intrinsic CLM evaluations, the form-meaning debate, and linguistic information content. 2 Related Work Multilingual Language Modeling. A CLM ( \mathcal{M} ) is trained to model the distribution of a token w_{i} conditioned on its preceding context: \text{NLL}(\mathcal{M})=-\sum_{i=1}^{S}\log P(w_{i}|w_{1},\dots,w_{i-1};\mathcal{M}). (1) The model has to maximize the log-probability, or equivalently, minimize the negative log-likelihood (NLL), of the cross-entropy of the model’s distribution and the true distribution of the next token. As mentioned, multilinguality for CLMs can be tackled from two angles: a multilingual model that includes many languages ( Scao et al. 2022 ; Üstün et al. 2024 ; Chang et al. 2024a , e.g., ) or comparing multiple monolingual models ( Mielke et al. 2019 ; Park et al. 2021 ; Chang et al. 2024b , e.g., ) . Our study is closer to the second, although we discuss implications for both. Intrinsic Evaluation. Intrinsic metrics for CLMs are based on how well a model predicts tokens in an unseen test sequence; perplexity being the most common. In order to make the evaluation fair, multi-parallel data is used. However, this type of data can have issues, such as translation effects or other artifacts. This has resulted in a number of slight variations or transformations of NLL that attempt to address these issues ( Cotterell et al. 2018 ; Wan 2022 ; Tsvetkov and Kipnis 2024 , e.g., ) . We discuss the metrics in detail in § 3.1 . Consistency. The ability of language models to equivalently handle paraphrases is known as consistency ( Elazar et al. 2021 ; Ohmer et al. 2024 , e.g., ) . Recently, Poelman et al. 2025 noted that comparing intrinsic metrics in multilingual settings could be unreliable due to being dependent on the form of parallel data. We repeat their hypothetical example in Table 1 . Model Sequence PPL A Sabe jugar al ajedrez 20 B Do you know how to play chess 22 B Can you play chess 18 Table 1: Two valid parallel English sentences and a Spanish sentence with hypothetical PPLs. If we select the first parallel English option, model A is “better” than B, and vice-versa for the second. Linguistic Universals. Instead of intrinsic metrics, we can also sample text from a CLM to evaluate how close it is to human language. Human languages seem to have a distinct ‘‘statistical fingerprint’’. 1 1 1 We refer to Bentz 2023 for recent empirical work and to Sproat 2023 for a broader overview. Perhaps the most famous metric related to this fingerprint is Zipf 1949 ’s law ( Zipf 1949 ): words in a corpus are inversely proportional to their frequency rank. This has been shown to hold across a number of human languages Piantadosi 2014 . The type-token ratio ( Herdan 1960 , or Heaps’ law;) is another law that characterizes vocabulary growth.