Information Locality as an Inductive Bias for Neural Language Models
Taiga Someya Anej Svete Brian DuSell, Timothy J. O’Donnell Mario Giulianelli Ryan Cotterell · 2025
Read Information Locality as an Inductive Bias for Neural Language Models by Taiga Someya Anej Svete Brian DuSell, Timothy J. O’Donnell Mario Giulianelli Ryan Cotterell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05136.txt.
Description
Taiga Someya and co-authors ask whether language models, like people, find languages easier when each next symbol is well pinned down by the few symbols before it. Their 2025 paper frames the question with a new measure of local entropy. It scores how much the preceding run of symbols cuts the doubt about the next one.
The team tested the idea on bent versions of natural text and on toy languages built from finite-state machines. In both, languages with higher local entropy proved harder for two model types to learn, a transformer and a recurrent net. The authors read this as a shared lean: models, much like human learners, key on the local statistical weave of a language.
Catalog record
| CREATOR | Taiga Someya Anej Svete Brian DuSell, Timothy J. O’Donnell Mario Giulianelli Ryan Cotterell |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 102 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05136