Note on Selection Bias in Observational Estimates of Algorithmic Progress
Parker Whitfill · 2025
Read Note on Selection Bias in Observational Estimates of Algorithmic Progress by Parker Whitfill (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11033.txt.
Description
Parker Whitfill wrote a short note in 2025 questioning estimates of algorithmic progress in language models. A 2024 study tracked model loss against compute over time and inferred rising efficiency. Whitfill argues the method has a hidden flaw. If part of a model's quality is unobserved, builders may spend compute in response to that quality. The data then select themselves, and the progress estimate gets contaminated. The note makes one methodological point rather than offering new estimates.
Catalog record
| CREATOR | Parker Whitfill |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-11033