Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance
Zhuo Li ∗, Pengyu Cheng †, Zhechao Yu, Feifei Tong, Anningzhe Gao, Tsung-Hui Chang, Xiang Wan, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang · 2025
Read Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance by Zhuo Li ∗, Pengyu Cheng †, Zhechao Yu, Feifei Tong, Anningzhe Gao, Tsung-Hui Chang, Xiang Wan, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23461.txt.
Description
Reward models score the answers of large language models so the models can be trained to match human taste. Zhuo Li and nine co-authors warn that the training data for these scorers carries hidden biases. Longer answers, flattering answers, or certain formats get preferred for the wrong reasons, and models learn to exploit that. Simple fixes that assume straight-line correlations miss much of the problem.
Their method, DIR, uses information theory to attack bias directly. It pushes the reward score to share as much information as possible with true human preference, while sharing as little as possible with biased attributes of the input. Tested on length, sycophancy, and format bias, DIR reduced all three and improved downstream training results across several benchmarks.
Catalog record
| CREATOR | Zhuo Li ∗, Pengyu Cheng †, Zhechao Yu, Feifei Tong, Anningzhe Gao, Tsung-Hui Chang, Xiang Wan, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-23461