Chunky Post-Training: Data Driven Failures of Generalization

Seoirse Murray, Allison Qi, Timothy Qian, John Schulman, Collin Burns, Sara Price · 2026

Read Chunky Post-Training: Data Driven Failures of Generalization by Seoirse Murray, Allison Qi, Timothy Qian, John Schulman, Collin Burns, Sara Price (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05910.txt.

Description

Post training teaches language models many behaviors from many data sets, and each data set smuggles in incidental patterns beside the intended lesson. Seoirse Murray, Allison Qi, and 4 co-authors call the result chunky post training: the model latches onto correlations between format and content, or narrow phrasings, and then surprises its creators. One quirk is rejecting true facts in a particular question format.

The team introduces two tools. SURF is a black box pipeline that surfaces unintended behaviors at run time. TURF traces a surfaced failure back to the specific post training data behind it. Applied to frontier models from several labs and to an open model family, the tools found miscalibrated behaviors rooted in imbalanced or underspecified data chunks. The paper argues that post training data curation, not just model design, drives many alignment surprises. It gives alignment teams a way to assign blame to data, precisely.

Catalog record

CREATORSeoirse Murray, Allison Qi, Timothy Qian, John Schulman, Collin Burns, Sara Price
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05910

Browse the full library · About the Tower of Babel