Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning
Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu · 2026
Read Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning by Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03328.txt.
Description
Xu and co-authors rethink how to pick calibration data for pruning large language models. Pruning at high sparsity leans on a small calibration set, often picked by habit. It is judged by perplexity or by mean scores on general tests. That one-axis view hides skill splits. At 60 percent sparsity, two plans close on commonsense scores can split by about 52 points on code retention.
The team treats calibration as a coverage problem across skills. A uniform four-way mix, called Balanced, beats every single source tested. Its edge over a standard corpus grows as sparsity rises. The edge holds across model sizes and a second model family. When aligned corpora are missing, a method called IGSP fakes the mix from the base model and the test taxonomy. It beat past self-generation methods by wide margins. On one model it topped Balanced itself.
Catalog record
| CREATOR | Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03328