Entropy in Large Language Models

Marco Scharringhausen · 2026

Read Entropy in Large Language Models by Marco Scharringhausen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-20052.txt.

Description

Language models produce text one symbol at a time. That makes their output a symbol source of the kind information theory studies. A 2026 arXiv paper by Marco Scharringhausen, working alone, treats model output as a stationary source over a finite alphabet and measures its entropy per word.

The comparison set is natural language, written and spoken, drawn from the Open American National Corpus. Model word entropy comes out lower than natural speech entropy in both written and spoken form. The longer-term aim is to make precise our intuitions about information and uncertainty in large language training. That includes judging what happens when models train on text other models generated, especially text scraped from the world wide web.

Catalog record

CREATORMarco Scharringhausen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-20052

Browse the full library · About the Tower of Babel