Intrinsic Entropy of Context Length Scaling in LLMs

Jingzhe Shi, Qinwei Ma, Hongyi Liu, Hang Zhao^, Jeng Neng Hwang, Lei Li^ · 2025

Read Intrinsic Entropy of Context Length Scaling in LLMs by Jingzhe Shi, Qinwei Ma, Hongyi Liu, Hang Zhao^, Jeng Neng Hwang, Lei Li^ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01481.txt.

Description

Jingzhe Shi and co-authors explain how context length affects language models in a 2025 paper. Long-context models can use huge inputs, but results are mixed: stray long context can hurt, while useful long context lowers error in patterned ways. The team wanted a single idea that explains both.

They propose intrinsic entropy as that measure and build a theory around it. Tests on natural text and on synthetic data backed their assumptions. One practical lesson is that the size of the training set sets an ideal context length, and in some cases caps how far scaling context can help. The authors hope the framework guides future model design and deeper study of how these models work.

Catalog record

CREATORJingzhe Shi, Qinwei Ma, Hongyi Liu, Hang Zhao^, Jeng Neng Hwang, Lei Li^
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-01481

Browse the full library · About the Tower of Babel