mSFT : Addressing Dataset Mixtures Overfitting Heterogeneously in Multi-task SFT

Woosung Koh, Jeyoung Jeon, Youngjin Song, Yujin Cheon, Soowon Oh, Jaehyeong Choi, Se-Young Yun · 2026

Read mSFT : Addressing Dataset Mixtures Overfitting Heterogeneously in Multi-task SFT by Woosung Koh, Jeyoung Jeon, Youngjin Song, Yujin Cheon, Soowon Oh, Jaehyeong Choi, Se-Young Yun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21606.txt.

Description

Koh and co-authors fix an imbalance in multi-task fine-tuning of language models. A single compute budget for every sub-dataset lets fast-learning tasks overfit while slow ones stay undertrained. Their mSFT algorithm trains on an active mixture, spots the earliest overfitting sub-dataset, and drops it. Training then resumes from that dataset's best checkpoint. Across ten benchmarks and six base models, mSFT beat four baselines. Gains held across dataset sizes and task mixes, and at low budgets mSFT even saved compute while improving scores.

Catalog record

CREATORWoosung Koh, Jeyoung Jeon, Youngjin Song, Yujin Cheon, Soowon Oh, Jaehyeong Choi, Se-Young Yun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52513
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21606

Browse the full library · About the Tower of Babel