Less is more: Not all samples are effective for evaluation

Wentang Song, Jinqiang Li, Kele Huang, Junhui Lin, Shengxiang Wu, Zhongshi Xie · 2026

Read Less is more: Not all samples are effective for evaluation by Wentang Song, Jinqiang Li, Kele Huang, Junhui Lin, Shengxiang Wu, Zhongshi Xie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03272.txt.

Introduction

Wentang Song and five co-authors argue that evaluating language models on every test sample is wasteful, and offer a way to shrink test sets without any history. Benchmarks for specialized domains carry heavy semantic redundancy and high evaluation cost. Existing compression methods need correctness labels from many models run on the full set, which fails in cold-start settings such as a brand-new task.

Their history-free framework first fine-tunes a base model on a small amount of domain data, then embeds every test sample in that adapted space using only its raw text. Task-aware clustering follows, with a dataset X-ray mechanism that reads cluster geometry to judge which samples carry real information. The goal is a compressed test set that still measures what the full one does.

Catalog record

CREATORWentang Song, Jinqiang Li, Kele Huang, Junhui Lin, Shengxiang Wu, Zhongshi Xie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03272

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Less is more: Not all samples are effective for evaluation Wentang Song, Jinqiang Li, Kele Huang, Junhui Lin, Shengxiang Wu, Zhongshi Xie Abstract The versatility of Large Language Models (LLMs) in vertical domains has spurred the development of numerous specialized evaluation benchmarks. However, these benchmarks often suffer from significant semantic redundancy and impose high computational costs during evaluation. Existing compression methods—such as tinyBenchmarks—depend critically on “correctness” labels from multiple historical models evaluated on the full test set, making them inapplicable in cold-start scenarios, such as the introduction of a new task, domain, or model with no prior evaluation history. To address this limitation, we propose a history-free test set compression framework that requires no prior model performance data. Our method begins by fine-tuning a base LLM on a small amount of domain-specific data to internalize task-relevant semantics. It then generates high-level semantic embeddings for all original test samples using only their raw textual content. In this domain-adapted embedding space, we perform task-aware clustering and introduce a novel “dataset X-ray” mechanism that analyzes cluster geometry to dynamically calibrate the compression intensity based on the intrinsic redundancy of the benchmark. Experiments on professional-domain datasets—notably a large-scale 3GPP communications benchmark—demonstrate that our approach effectively identifies and removes redundant samples, reducing evaluation cost by over 90% while preserving high fidelity to the full benchmark. † † address: ZTE 1 Introduction Large Language Models (LLMs) represent a transformative advance in artificial intelligence, demonstrating unprecedented capabilities in understanding and generating human-like language [ 1 ] . Trained on vast corpora spanning diverse linguistic patterns, modern LLMs excel across a wide spectrum of tasks—from factual question answering and logical reasoning [ 3 ] to creative writing and domain-specific problem solving [ 5 ] . Their potential to revolutionize fields such as education, finance, telecommunications, and healthcare has spurred rapid development of specialized models tailored to vertical domains. However, as LLMs grow in scale and specialization, the challenge of efficiently and reliably evaluating their capabilities becomes increasingly acute [ 12 ] . Current evaluation practices rely heavily on large-scale benchmarks—often comprising tens of thousands of items—designed to assess a model’s proficiency across multiple dimensions [ 8 , 4 ] . Yet these benchmarks frequently suffer from high redundancy: many questions probe overlapping skills or knowledge, leading to unnecessary computational overhead without proportional gains in diagnostic insight [ 2 , 7 ] . This inefficiency is particularly problematic in resource-constrained settings or during rapid model iteration cycles. To address this, recent works such as tinyBenchmarks [ 11 ] and metabench [ 6 ] propose compressing existing benchmarks into smaller, more informative subsets. While effective, these approaches share a critical limitation: they depend on historical evaluation data—specifically, the performance profiles of numerous pre-existing models on the full benchmark. This reliance renders them inapplicable in cold-start scenarios, such as: (i) evaluating a newly developed domain-specific model with no prior evaluation history; (ii) constructing a benchmark for an emerging field (e.g., 6G protocol validation) where no reference models exist; or (iii) assessing open-ended generative tasks lacking definitive “correctness” labels. To overcome this fundamental bottleneck, we propose a self-contained, history-free test set compression framework for domain-adapted LLMs. Our core insight is that a model fine-tuned on domain-specific data inherently internalizes the semantic structure and knowledge topology of that domain. This internal representation can be leveraged—without any external model feedback—to identify and eliminate redundant samples directly from the raw test set. Our method operates in three stages: (1) fine-tune a LLM on a small amount of domain data; (2) generate high-level semantic embeddings for all test samples using the adapted model; and (3) perform task-aware clustering in the embedding space to select a diverse, representative subset. We further introduce an “dataset X-ray” mechanism that analyzes cluster geometry to dynamically adjust compression intensity based on dataset redundancy. This approach enables rapid, low-cost evaluation of new domain models with no dependency on historical data, while preserving high fidelity to the original benchmark. Moreover, the resulting compressed set serves as high-quality seed data for generating generalization samples to detect test-set contamination—a growing concern in LLM evaluation. By decoupling compression from historical model behavior and grounding it in intrinsic semantic structure, our framework offers a practical, scalable solution for the next generation of specialized LLM assessment. 2 Methodology Figure 1: The pipeline of our proposed method. In this paper, we propose a history-free, domain-adaptive test set compression framework for large language models (LLMs). Our approach leverages the semantic understanding of a domain-finetuned LLM to construct a compact yet representative evaluation subset directly from the raw test set, without requiring any historical model predictions or ground-truth correctness labels. The overall pipeline is illustrated in Figure 1 and consists of three core stages: domain adaptation via fine-tuning, semantic embedding generation, and task-aware adaptive sampling. 2.1 Domain Adaptation via Full-Parameter Fine-Tuning To endow the LLM with domain-specific knowledge, we perform full-parameter fine-tuning on a small set of labeled data from the target domain (e.g., 3GPP communication protocols). Let D_{domain}=\{(x_{i},y_{i})\}_{i=1}^{N} denote the domain training set, where x_{i} is a textual input (e.g., a protocol description or troubleshooting query) and y_{i} is its corresponding label or reference answer. We initialize the model with a pre-trained LLM (e.g., Qwen-1.7B [ 13 ] ) and fine-tune all parameters using standard cross-entropy loss: \mathcal{L}_{\mathrm{ft}}=-\sum_{i=1}^{N}\log P(y_{i}\mid\mathbf{x}_{i};\theta) where \theta denotes the model parameters. The resulting domain-adapted model f_{\theta\*} internalizes the terminologies, logical structures, and task semantics of the target domain, enabling high-fidelity semantic representation of test samples. 2.2 Semantic Embedding Generation Given the original test set \mathcal{T}=\{\mathbf{q}_{j}\}_{j=1}^{M} , where each qj is a raw question or instruction (e.g., “What is the cause of RLF during handover?”), we generate a semantic embedding for each sample using the domain-adapted model. Specifically, we format each query with a domain-specific system prompt (e.g., “You are a 5G communication expert.”) and feed it into f_{\theta\*} as: messages=[\{role:"system",content:"You\quad are\quad a\quad 5G\quad communication\quad expert."\},\{role:"user", content:q_{j}\}]. We extract the final hidden state of the last Transformer layer as the embedding vector e_{j}\in{R^{d}} . To enable reliable similarity computation, we apply L2 normalization to each embedding: \hat{\mathbf{e}}_{j}=\frac{\mathbf{e}_{j}}{\|\mathbf{e}_{j}\|_{2}} The resulting set \{\hat{\mathbf{e}}_{j}\}_{j=1}^{M} forms a task-aware semantic space where geometric proximity reflects semantic equivalence in the target domain. 2.3 Task-Aware Adaptive Sampling We compress the test set by selecting a subset \mathcal{T}_{comp}\subset\mathcal{T} that preserves the semantic coverage and task diversity of the original set. This is achieved via a two-stage sampling strategy: 2.3.1 Clustering-Based Redundancy Analysis We apply K-means clustering on \{\hat{\mathbf{e}}_{j}\} to partition the test set into K semantic groups. The clustering reveals the intrinsic redundancy structure: dense clusters indicate high semantic overlap, while sparse or isolated points signify unique task instances. 2.3.2 Dataset X-ray and Dynamic Sampling We introduce the Dataset X-ray mechanism to dynamically adjust sampling intensity based on cluster geometry: • If clusters are compact and well-separated (high redundancy), we perform aggressive sampling: within each cluster, we divide samples into 5 distance intervals (from centroid) and uniformly select 10% from each interval to ensure both core and boundary coverage. • If clusters are diffuse or overlapping (low redundancy), we reduce compression intensity (e.g., retain 20–30% of samples) to avoid losing critical task dimensions. This strategy ensures that \mathcal{T}_{comp} maintains balanced representation across knowledge areas while

Read the complete text