mSFT : Addressing Dataset Mixtures Overfitting Heterogeneously in Multi-task SFT
Woosung Koh, Jeyoung Jeon, Youngjin Song, Yujin Cheon, Soowon Oh, Jaehyeong Choi, Se-Young Yun · 2026
Read mSFT : Addressing Dataset Mixtures Overfitting Heterogeneously in Multi-task SFT by Woosung Koh, Jeyoung Jeon, Youngjin Song, Yujin Cheon, Soowon Oh, Jaehyeong Choi, Se-Young Yun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21606.txt.
Description
Koh and co-authors fix an imbalance in multi-task fine-tuning of language models. A single compute budget for every sub-dataset lets fast-learning tasks overfit while slow ones stay undertrained. Their mSFT algorithm trains on an active mixture, spots the earliest overfitting sub-dataset, and drops it. Training then resumes from that dataset's best checkpoint. Across ten benchmarks and six base models, mSFT beat four baselines. Gains held across dataset sizes and task mixes, and at low budgets mSFT even saved compute while improving scores.
Catalog record
| CREATOR | Woosung Koh, Jeyoung Jeon, Youngjin Song, Yujin Cheon, Soowon Oh, Jaehyeong Choi, Se-Young Yun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52513 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21606