Iterative Finetuning is Mostly Idempotent

Zephaniah Roe, Jack Sanderson, Dang Nguyen, Julian Huang Todd Nief Aryan Shrivastava Chenhao Tan Ari Holtzman · 2026

Read Iterative Finetuning is Mostly Idempotent by Zephaniah Roe, Jack Sanderson, Dang Nguyen, Julian Huang Todd Nief Aryan Shrivastava Chenhao Tan Ari Holtzman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01130.txt.

Description

Zephaniah Roe and seven co-authors ask whether training a model on its own outputs amplifies behavior traits such as sycophancy or misalignment across generations. They train series of models where each is finetuned on data generated by its predecessor, starting from a model seeded with a persona or belief. Three settings are tested: supervised tuning, synthetic document finetuning, and direct preference optimization.

In the two other settings, traits mostly decay or stay constant, so further cycles do little. Rare amplification generally costs coherence. In the preference-based one, the effect can build reliably with continual training favoring a model's own outputs. It vanishes when models are reinitialized each cycle. Limiting repeated post-training may be an effective defense.

Catalog record

CREATORZephaniah Roe, Jack Sanderson, Dang Nguyen, Julian Huang Todd Nief Aryan Shrivastava Chenhao Tan Ari Holtzman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01130

Browse the full library · About the Tower of Babel