How Should We Model the Probability of a Language?

Rasul Dent, Pedro Ortiz Suarez, Thibault Clérice, Benoît Sagot · 2026

Read How Should We Model the Probability of a Language? by Rasul Dent, Pedro Ortiz Suarez, Thibault Clérice, Benoît Sagot (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08951.txt.

Description

Of more than 7,000 spoken languages, commercial identifiers reliably recognize only a few hundred in writing. Rasul Dent and three co-authors argue this gap is largely self-inflicted. The field treats language identification as context-free text classification, hiding the central role of prior probabilities, and institutions reward global models with fixed priors.

This position paper reframes identification as a routing problem: given where and how text appears, which languages are locally plausible? It calls for principled ways to fold environmental cues into the decision, so tail languages get a fair chance. Better priors, not just bigger classifiers, are the path to wider coverage.

Catalog record

CREATORRasul Dent, Pedro Ortiz Suarez, Thibault Clérice, Benoît Sagot
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-08951

Browse the full library · About the Tower of Babel