SPICE: Submodular Penalized Information–Conflict Selection for Efficient Large Language Model Training

Powei Chang Jinpeng Zhang Bowen Chen Chenyu Wang Chenlu Guo Yixing Zhang Yukang Gao JianXiang Xiang Yue Gao Chaoqun Sun Yiyi Chen Dongying Kong Bilibili Inc. {zhangbowei01,zhangjinpeng01,chenbowen,wangchenyu06,guochenlu,zhangyixing,gaoyukang, xiangjianxiang,gaoyue02,sunchaoqun,chenyiyi,kongdongying} · 2026

Read SPICE: Submodular Penalized Information–Conflict Selection for Efficient Large Language Model Training by Powei Chang Jinpeng Zhang Bowen Chen Chenyu Wang Chenlu Guo Yixing Zhang Yukang Gao JianXiang Xiang Yue Gao Chaoqun Sun Yiyi Chen Dongying Kong Bilibili Inc. {zhangbowei01,zhangjinpeng01,chenbowen,wangchenyu06,guochenlu,zhangyixing,gaoyukang, xiangjianxiang,gaoyue02,sunchaoqun,chenyiyi,kongdongying} (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23155.txt.

Description

Powei Chang and a large team select training data for language models using information theory. Picking examples that maximize the log-determinant of Fisher information is attractive, because that objective is submodular. Submodularity lets greedy algorithms provably reach a large share of the best possible gain under a size budget.

Their SPICE method adds penalized conflict handling to that selection. The goal is efficient training: fewer, better-chosen examples for the same final quality. Careful data choice can cut the cost of instruction tuning sharply. A cardinality budget keeps selection honest: the method must choose a fixed-size set, not quietly use more data.

Catalog record

CREATORPowei Chang Jinpeng Zhang Bowen Chen Chenyu Wang Chenlu Guo Yixing Zhang Yukang Gao JianXiang Xiang Yue Gao Chaoqun Sun Yiyi Chen Dongying Kong Bilibili Inc. {zhangbowei01,zhangjinpeng01,chenbowen,wangchenyu06,guochenlu,zhangyixing,gaoyukang, xiangjianxiang,gaoyue02,sunchaoqun,chenyiyi,kongdongying}
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-23155

Browse the full library · About the Tower of Babel