Information Locality as an Inductive Bias for Neural Language Models

Taiga Someya Anej Svete Brian DuSell, Timothy J. O’Donnell Mario Giulianelli Ryan Cotterell · 2025

Read Information Locality as an Inductive Bias for Neural Language Models by Taiga Someya Anej Svete Brian DuSell, Timothy J. O’Donnell Mario Giulianelli Ryan Cotterell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05136.txt.

Description

Taiga Someya and co-authors ask whether language models, like people, find languages easier when each next symbol is well pinned down by the few symbols before it. Their 2025 paper frames the question with a new measure of local entropy. It scores how much the preceding run of symbols cuts the doubt about the next one.

The team tested the idea on bent versions of natural text and on toy languages built from finite-state machines. In both, languages with higher local entropy proved harder for two model types to learn, a transformer and a recurrent net. The authors read this as a shared lean: models, much like human learners, key on the local statistical weave of a language.

Catalog record

CREATORTaiga Someya Anej Svete Brian DuSell, Timothy J. O’Donnell Mario Giulianelli Ryan Cotterell
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE102 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05136

Browse the full library · About the Tower of Babel