DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data

Yuhang Zhou, Jing Zhu, Shengyi Qian, Zhuokai Zhao, Xiyao Wang, Xiaoyu Liu, Ming Li, Paiheng Xu, Wei Ai, Furong Huang · 2025

Read DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data by Yuhang Zhou, Jing Zhu, Shengyi Qian, Zhuokai Zhao, Xiyao Wang, Xiaoyu Liu, Ming Li, Paiheng Xu, Wei Ai, Furong Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15074.txt.

Description

Yuhang Zhou and co-authors fix a fairness flaw in how language models learn from feedback in this 2025 paper. A popular training method compares groups of answers, but it quietly assumes training topics are evenly spread. Real data skews hard toward common topics, so rare ones get neglected.

Their method, DISCO, adjusts rewards in two ways. A domain-aware scaling reweights each topic by its share of the data, calming the frequency bias. A difficulty-aware scaling uses the model's own consistency to find murky prompts worth extra attention. Across several models and skewed datasets, the method beat its parent approach and set new marks on multi-domain tests. Code and data are public.

Catalog record

CREATORYuhang Zhou, Jing Zhu, Shengyi Qian, Zhuokai Zhao, Xiyao Wang, Xiaoyu Liu, Ming Li, Paiheng Xu, Wei Ai, Furong Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15074

Browse the full library · About the Tower of Babel