Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning

Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu · 2026

Read Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning by Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03328.txt.

Description

Xu and co-authors rethink how to pick calibration data for pruning large language models. Pruning at high sparsity leans on a small calibration set, often picked by habit. It is judged by perplexity or by mean scores on general tests. That one-axis view hides skill splits. At 60 percent sparsity, two plans close on commonsense scores can split by about 52 points on code retention.

The team treats calibration as a coverage problem across skills. A uniform four-way mix, called Balanced, beats every single source tested. Its edge over a standard corpus grows as sparsity rises. The edge holds across model sizes and a second model family. When aligned corpora are missing, a method called IGSP fakes the mix from the base model and the test taxonomy. It beat past self-generation methods by wide margins. On one model it topped Balanced itself.

Catalog record

CREATORHu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03328

Browse the full library · About the Tower of Babel