Intrinsic Entropy of Context Length Scaling in LLMs
Jingzhe Shi, Qinwei Ma, Hongyi Liu, Hang Zhao^, Jeng Neng Hwang, Lei Li^ · 2025
Read Intrinsic Entropy of Context Length Scaling in LLMs by Jingzhe Shi, Qinwei Ma, Hongyi Liu, Hang Zhao^, Jeng Neng Hwang, Lei Li^ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01481.txt.
Description
Jingzhe Shi and co-authors explain how context length affects language models in a 2025 paper. Long-context models can use huge inputs, but results are mixed: stray long context can hurt, while useful long context lowers error in patterned ways. The team wanted a single idea that explains both.
They propose intrinsic entropy as that measure and build a theory around it. Tests on natural text and on synthetic data backed their assumptions. One practical lesson is that the size of the training set sets an ideal context length, and in some cases caps how far scaling context can help. The authors hope the framework guides future model design and deeper study of how these models work.
Catalog record
| CREATOR | Jingzhe Shi, Qinwei Ma, Hongyi Liu, Hang Zhao^, Jeng Neng Hwang, Lei Li^ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-01481