Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization

Andy Zeyi Liu, Elliot Paquette, John Sous · 2026

Read Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization by Andy Zeyi Liu, Elliot Paquette, John Sous (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05683.txt.

Description

Loss curves say a language model is improving; they say little about what is forming inside. Andy Zeyi Liu, Elliot Paquette, and John Sous propose spectral diagnostics for that hidden mechanics. Using a controlled family of small decoder models, they measure activation covariance spectra and per-sample gradient spectra throughout training.

Three findings stand out. Batch size quietly determines representation geometry: runs reaching equal loss settle into distinct spectra. Early covariance tails forecast later token efficiency. And the gradient view explains the pattern mechanistically. The protocol turns spectra into practical instruments for steering optimization.

Catalog record

CREATORAndy Zeyi Liu, Elliot Paquette, John Sous
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05683

Browse the full library · About the Tower of Babel