Rethinking Domain-Specific LLM Benchmark Construction: A Comprehensiveness-Compactness Approach
Rubing Chen, Jiaxin Wu, Jian Wang, Xulu Zhang, Wenqi Fan, Chenghua Lin, Xiao-Yong Wei, Qing Li · 2025
Read Rethinking Domain-Specific LLM Benchmark Construction: A Comprehensiveness-Compactness Approach by Rubing Chen, Jiaxin Wu, Jian Wang, Xulu Zhang, Wenqi Fan, Chenghua Lin, Xiao-Yong Wei, Qing Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07353.txt.
Description
Rubing Chen and co-authors argue that bigger is not always better when building benchmarks for specialized language models. Their 2025 paper observes that domain benchmarks usually chase coverage by piling up huge corpora and question sets. Yet nobody had checked how that design choice affects the precision and recall of the scores produced.
They propose Comp-Comp, an iterative framework built on two aims: cover the whole domain, but keep the set compact and free of redundancy. A case study at a well-known university produced PolyBench, a large academic benchmark built under these rules. Tests suggest the approach measures domain skill more cleanly than raw scaling. Though the case is academic, the authors say the framework transfers to any specialty. Code and data are released for reuse.
Catalog record
| CREATOR | Rubing Chen, Jiaxin Wu, Jian Wang, Xulu Zhang, Wenqi Fan, Chenghua Lin, Xiao-Yong Wei, Qing Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-07353