Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
Jacob Dineen, Aswin RRV, Zhikun Xu, Ben Zhou · 2026
Read Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution by Jacob Dineen, Aswin RRV, Zhikun Xu, Ben Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03472.txt.
Description
Dineen and colleagues tackle a stall in co-evolutionary self-play, in which one language model writes problems and another solves them. The problem-writer soon narrows to a thin set of problems, so the solver stops learning. Their fix, vocabulary dropout, randomly masks the writer's output scores during training and problem generation. Tests on Qwen3 math reasoning show the writer stays varied and the solver gains about 4.4 points at 8B size.
Catalog record
| CREATOR | Jacob Dineen, Aswin RRV, Zhikun Xu, Ben Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57036 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03472