Rethinking Domain-Specific LLM Benchmark Construction: A Comprehensiveness-Compactness Approach

Rubing Chen, Jiaxin Wu, Jian Wang, Xulu Zhang, Wenqi Fan, Chenghua Lin, Xiao-Yong Wei, Qing Li · 2025

Read Rethinking Domain-Specific LLM Benchmark Construction: A Comprehensiveness-Compactness Approach by Rubing Chen, Jiaxin Wu, Jian Wang, Xulu Zhang, Wenqi Fan, Chenghua Lin, Xiao-Yong Wei, Qing Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07353.txt.

Description

Rubing Chen and co-authors argue that bigger is not always better when building benchmarks for specialized language models. Their 2025 paper observes that domain benchmarks usually chase coverage by piling up huge corpora and question sets. Yet nobody had checked how that design choice affects the precision and recall of the scores produced.

They propose Comp-Comp, an iterative framework built on two aims: cover the whole domain, but keep the set compact and free of redundancy. A case study at a well-known university produced PolyBench, a large academic benchmark built under these rules. Tests suggest the approach measures domain skill more cleanly than raw scaling. Though the case is academic, the authors say the framework transfers to any specialty. Code and data are released for reuse.

Catalog record

CREATORRubing Chen, Jiaxin Wu, Jian Wang, Xulu Zhang, Wenqi Fan, Chenghua Lin, Xiao-Yong Wei, Qing Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07353

Browse the full library · About the Tower of Babel