Manifold Trajectories in Next-Token Prediction: From Replicator Dynamics to Softmax Equilibrium

Christopher R. Lee-Jenkins · 2025

Read Manifold Trajectories in Next-Token Prediction: From Replicator Dynamics to Softmax Equilibrium by Christopher R. Lee-Jenkins (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21186.txt.

Description

Lee-Jenkins re-derives how a language model picks its next word, and finds a familiar dynamic hiding inside. The standard picture is simple bookkeeping: score every candidate word, soften the scores, and normalize into probabilities. Lee-Jenkins shows that step can be read as a constrained optimization on the probability simplex, the space of all possible word distributions.

Done step by step, the update matches a classic multiplicative-weights rule. Smoothed into continuous time, it becomes the replicator flow known from evolutionary theory. Along that path, the distribution glides smoothly to the usual softmax answer. Temperature, he proves, merely rescales time along the same path, and top-k or nucleus sampling just fence the flow onto one face. He also sketches how path-shaped tweaks to scores might mimic hallucination-like loops.

Catalog record

CREATORChristopher R. Lee-Jenkins
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-21186

Browse the full library · About the Tower of Babel