How Should We Model the Probability of a Language?
Rasul Dent, Pedro Ortiz Suarez, Thibault Clérice, Benoît Sagot · 2026
Read How Should We Model the Probability of a Language? by Rasul Dent, Pedro Ortiz Suarez, Thibault Clérice, Benoît Sagot (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08951.txt.
Description
Of more than 7,000 spoken languages, commercial identifiers reliably recognize only a few hundred in writing. Rasul Dent and three co-authors argue this gap is largely self-inflicted. The field treats language identification as context-free text classification, hiding the central role of prior probabilities, and institutions reward global models with fixed priors.
This position paper reframes identification as a routing problem: given where and how text appears, which languages are locally plausible? It calls for principled ways to fold environmental cues into the decision, so tail languages get a fair chance. Better priors, not just bigger classifiers, are the path to wider coverage.
Catalog record
| CREATOR | Rasul Dent, Pedro Ortiz Suarez, Thibault Clérice, Benoît Sagot |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-08951