Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution

Jacob Dineen, Aswin RRV, Zhikun Xu, Ben Zhou · 2026

Read Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution by Jacob Dineen, Aswin RRV, Zhikun Xu, Ben Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03472.txt.

Description

Dineen and colleagues tackle a stall in co-evolutionary self-play, in which one language model writes problems and another solves them. The problem-writer soon narrows to a thin set of problems, so the solver stops learning. Their fix, vocabulary dropout, randomly masks the writer's output scores during training and problem generation. Tests on Qwen3 math reasoning show the writer stays varied and the solver gains about 4.4 points at 8B size.

Catalog record

CREATORJacob Dineen, Aswin RRV, Zhikun Xu, Ben Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57036
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03472

Browse the full library · About the Tower of Babel