Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing
Valentina Njaradi, Clémentine Dominé, Rachel Swanson, Marco Mondelli †, Andrew Saxe † · 2026
Read Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing by Valentina Njaradi, Clémentine Dominé, Rachel Swanson, Marco Mondelli †, Andrew Saxe † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20105.txt.
Description
Valentina Njaradi and four co-workers model pretraining and linear probing to find the optimal representation size. Structure spotting is framed as PCA on raw data, and downstream learning as linear fit on tagged data. In the many-feature regime, they derive training and test error in terms of representation size, sample sizes, and task fit. With pretraining data but little downstream data, the tightest are optimal, whereas scarce data favours larger ones. They show a pretraining-labels trade-off, counting raw data needed to replace one tagged sample.
Catalog record
| CREATOR | Valentina Njaradi, Clémentine Dominé, Rachel Swanson, Marco Mondelli †, Andrew Saxe † |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 158506 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20105