SPICE: Submodular Penalized Information–Conflict Selection for Efficient Large Language Model Training
Powei Chang Jinpeng Zhang Bowen Chen Chenyu Wang Chenlu Guo Yixing Zhang Yukang Gao JianXiang Xiang Yue Gao Chaoqun Sun Yiyi Chen Dongying Kong Bilibili Inc. {zhangbowei01,zhangjinpeng01,chenbowen,wangchenyu06,guochenlu,zhangyixing,gaoyukang, xiangjianxiang,gaoyue02,sunchaoqun,chenyiyi,kongdongying} · 2026
Read SPICE: Submodular Penalized Information–Conflict Selection for Efficient Large Language Model Training by Powei Chang Jinpeng Zhang Bowen Chen Chenyu Wang Chenlu Guo Yixing Zhang Yukang Gao JianXiang Xiang Yue Gao Chaoqun Sun Yiyi Chen Dongying Kong Bilibili Inc. {zhangbowei01,zhangjinpeng01,chenbowen,wangchenyu06,guochenlu,zhangyixing,gaoyukang, xiangjianxiang,gaoyue02,sunchaoqun,chenyiyi,kongdongying} (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23155.txt.
Description
Powei Chang and a large team select training data for language models using information theory. Picking examples that maximize the log-determinant of Fisher information is attractive, because that objective is submodular. Submodularity lets greedy algorithms provably reach a large share of the best possible gain under a size budget.
Their SPICE method adds penalized conflict handling to that selection. The goal is efficient training: fewer, better-chosen examples for the same final quality. Careful data choice can cut the cost of instruction tuning sharply. A cardinality budget keeps selection honest: the method must choose a fixed-size set, not quietly use more data.
Catalog record
| CREATOR | Powei Chang Jinpeng Zhang Bowen Chen Chenyu Wang Chenlu Guo Yixing Zhang Yukang Gao JianXiang Xiang Yue Gao Chaoqun Sun Yiyi Chen Dongying Kong Bilibili Inc. {zhangbowei01,zhangjinpeng01,chenbowen,wangchenyu06,guochenlu,zhangyixing,gaoyukang, xiangjianxiang,gaoyue02,sunchaoqun,chenyiyi,kongdongying} |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-23155