Escaping the Mode Lottery: Multi-Response Training Improves Language Model Generalization

Hasan Amin, Kian Ahrabian, Ming Yin, Rajiv Khanna · 2026

Read Escaping the Mode Lottery: Multi-Response Training Improves Language Model Generalization by Hasan Amin, Kian Ahrabian, Ming Yin, Rajiv Khanna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00544.txt.

Description

Amin and co-authors show that more than one good reply per prompt can lift model training. Stock fine-tune pairs each prompt with just one reply. Many prompts hold more than one sound end, though. That cut turns a rich spread to a one-draw view, a mode lottery in their words.

Their multi-reply training keeps more than one reply per prompt in the set. Prompts and replies act as split pools of proof in their math. More prompts cut doubt on what comes in. More replies cut doubt on what should go out. That trade calls when extra replies pay and when they fade. Pick rules for replies are weighed with care as well. Gains peak when replies are rich and prompts do not repeat.

Catalog record

CREATORHasan Amin, Kian Ahrabian, Ming Yin, Rajiv Khanna
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE129 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00544

Browse the full library · About the Tower of Babel