Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing

Valentina Njaradi, Clémentine Dominé, Rachel Swanson, Marco Mondelli †, Andrew Saxe † · 2026

Read Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing by Valentina Njaradi, Clémentine Dominé, Rachel Swanson, Marco Mondelli †, Andrew Saxe † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20105.txt.

Description

Valentina Njaradi and four co-workers model pretraining and linear probing to find the optimal representation size. Structure spotting is framed as PCA on raw data, and downstream learning as linear fit on tagged data. In the many-feature regime, they derive training and test error in terms of representation size, sample sizes, and task fit. With pretraining data but little downstream data, the tightest are optimal, whereas scarce data favours larger ones. They show a pretraining-labels trade-off, counting raw data needed to replace one tagged sample.

Catalog record

CREATORValentina Njaradi, Clémentine Dominé, Rachel Swanson, Marco Mondelli †, Andrew Saxe †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE158506
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20105

Browse the full library · About the Tower of Babel