Sigma : Scalable Spectral Insights for LLM Model Collapse

Yi Gu, Lingyou Pang, Xiangkun Ye, Tianyu Wang, Jianyu Lin, Carey E. Priebe, Alexander Aue · 2026

Read Sigma : Scalable Spectral Insights for LLM Model Collapse by Yi Gu, Lingyou Pang, Xiangkun Ye, Tianyu Wang, Jianyu Lin, Carey E. Priebe, Alexander Aue (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03385.txt.

Description

Yi Gu and six co-authors build SIGMA, a framework for measuring model collapse, the decay that sets in when language models are trained on their own generated text. Recursive training on synthetic data shrinks the variety and quality of what a model can represent, but rigorous ways to quantify or predict the onset in high dimensions have been missing. SIGMA watches the spectrum of the embedding Gram matrix, the table of similarities between representations.

The authors derive deterministic and stochastic bounds on that spectrum and turn them into a grounded metric for how much the representation space has contracted. The stochastic version estimates the bounds without full eigendecomposition, so the method scales to large foundation models. Demonstrations show SIGMA tracking the transition toward degeneracy, offering an early-warning gauge for training pipelines that recycle synthetic data.

Catalog record

CREATORYi Gu, Lingyou Pang, Xiangkun Ye, Tianyu Wang, Jianyu Lin, Carey E. Priebe, Alexander Aue
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03385

Browse the full library · About the Tower of Babel