Iterative Finetuning is Mostly Idempotent
Zephaniah Roe, Jack Sanderson, Dang Nguyen, Julian Huang Todd Nief Aryan Shrivastava Chenhao Tan Ari Holtzman · 2026
Read Iterative Finetuning is Mostly Idempotent by Zephaniah Roe, Jack Sanderson, Dang Nguyen, Julian Huang Todd Nief Aryan Shrivastava Chenhao Tan Ari Holtzman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01130.txt.
Description
Zephaniah Roe and seven co-authors ask whether training a model on its own outputs amplifies behavior traits such as sycophancy or misalignment across generations. They train series of models where each is finetuned on data generated by its predecessor, starting from a model seeded with a persona or belief. Three settings are tested: supervised tuning, synthetic document finetuning, and direct preference optimization.
In the two other settings, traits mostly decay or stay constant, so further cycles do little. Rare amplification generally costs coherence. In the preference-based one, the effect can build reliably with continual training favoring a model's own outputs. It vanishes when models are reinitialized each cycle. Limiting repeated post-training may be an effective defense.
Catalog record
| CREATOR | Zephaniah Roe, Jack Sanderson, Dang Nguyen, Julian Huang Todd Nief Aryan Shrivastava Chenhao Tan Ari Holtzman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01130