On the Asymptotics of Self-Supervised Pre-training: Two-Stage M M -Estimation and Representation Symmetry

Mohammad Tinati, Stephen Tu · 2026

Read On the Asymptotics of Self-Supervised Pre-training: Two-Stage M M -Estimation and Representation Symmetry by Mohammad Tinati, Stephen Tu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27631.txt.

Introduction

Tinati and Tu build limit theory for pretraining from unlabeled data. Models learn features first, then fine-tune for a task. Learned features are fixed only up to a symmetry. That twist breaks plain two-stage math. Tools from curved geometry handle the core quantities instead. A link through orbit sameness ties them to test risk. The team pins down the limit law for that risk. Case studies cover spectral methods, factor models, and mixtures.

Catalog record

CREATORMohammad Tinati, Stephen Tu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE250512
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27631

Browse the full library · About the Tower of Babel