Repeated Sequences Reveal Gaps between Large Language Models and Natural Language

Kumiko Tanaka-Ishii · 2026

Read Repeated Sequences Reveal Gaps between Large Language Models and Natural Language by Kumiko Tanaka-Ishii (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24850.txt.

Description

Kumiko Tanaka-Ishii proposes repeated subsequences as a diagnostic separating language models from natural language. Whether models capture language structure beyond local fluency remains open, since task scores reveal little about long-range organization. Analyzing how texts reuse earlier structure across scales, and relating reuse to Renyi entropies, probes finite-length organization. Power-law models fit restricted block-length ranges, but entropy growth is often better described by logarithmic-power forms. Natural language shows stable entropy-growth patterns, while GPT texts show clear, size-linked shifts in estimated exponents.

Catalog record

CREATORKumiko Tanaka-Ishii
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55592
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24850

Browse the full library · About the Tower of Babel