Chunky Post-Training: Data Driven Failures of Generalization
Seoirse Murray, Allison Qi, Timothy Qian, John Schulman, Collin Burns, Sara Price · 2026
Read Chunky Post-Training: Data Driven Failures of Generalization by Seoirse Murray, Allison Qi, Timothy Qian, John Schulman, Collin Burns, Sara Price (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05910.txt.
Description
Post training teaches language models many behaviors from many data sets, and each data set smuggles in incidental patterns beside the intended lesson. Seoirse Murray, Allison Qi, and 4 co-authors call the result chunky post training: the model latches onto correlations between format and content, or narrow phrasings, and then surprises its creators. One quirk is rejecting true facts in a particular question format.
The team introduces two tools. SURF is a black box pipeline that surfaces unintended behaviors at run time. TURF traces a surfaced failure back to the specific post training data behind it. Applied to frontier models from several labs and to an open model family, the tools found miscalibrated behaviors rooted in imbalanced or underspecified data chunks. The paper argues that post training data curation, not just model design, drives many alignment surprises. It gives alignment teams a way to assign blame to data, precisely.
Catalog record
| CREATOR | Seoirse Murray, Allison Qi, Timothy Qian, John Schulman, Collin Burns, Sara Price |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-05910