Better Datasets Start From RefineLab: Automatic Optimization for High-Quality Dataset Refinement

Xiaonan Luo ∗, Yue Huang ∗, Ping He, Xiangliang Zhang · 2025

Read Better Datasets Start From RefineLab: Automatic Optimization for High-Quality Dataset Refinement by Xiaonan Luo ∗, Yue Huang ∗, Ping He, Xiangliang Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06530.txt.

Introduction

Xiaonan Luo, Yue Huang, Ping He, and Xiangliang Zhang present RefineLab, a system that polishes question-answer datasets on its own. Good test sets are the bedrock of trustworthy language model testing, yet even expert-made sets show gaps in coverage, skewed difficulty, and factual slips. Machine-generated sets have made those quality problems worse.

RefineLab treats refinement as a budget problem: spend a fixed token allowance where edits help most. Given target qualities such as coverage and difficulty balance plus a menu of edits like rephrasing, an assignment module picks the best operation per question. Results narrowed the gap to expert datasets on coverage, difficulty, factual fidelity, and distractor quality.

Catalog record

CREATORXiaonan Luo ∗, Yue Huang ∗, Ping He, Xiangliang Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06530

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Better Datasets Start From RefineLab: Automatic Optimization for High-Quality Dataset Refinement Xiaonan Luo ∗, Yue Huang ∗, Ping He, Xiangliang Zhang Abstract High - quality Question–Answer (QA) datasets are foundational for reliable Large Language Model (LLM) evaluation, yet even expert - crafted datasets exhibit persistent gaps in domain coverage, misaligned difficulty distributions, and factual inconsistencies. The recent surge in generative model-powered datasets has compounded these quality challenges. In this work, we introduce RefineLab , the first LLM - driven framework that automatically refines raw QA textual data into high-quality datasets under a controllable token - budget constraint. RefineLab takes a set of target quality attributes (such as coverage and difficulty balance) as refinement objectives, and performs selective edits within a predefined token budget to ensure practicality and efficiency. In essence, RefineLab addresses a constrained optimization problem: improving the quality of QA samples as much as possible while respecting resource limitations. With a set of available refinement operations (e.g., rephrasing, distractor replacement), RefineLab takes as input the original dataset, a specified set of target quality dimensions, and a token budget, and determines which refinement operations should be applied to each QA sample. This process is guided by an assignment module that selects optimal refinement strategies to maximize overall dataset quality while adhering to the budget constraint. Experiments demonstrate that RefineLab consistently narrows divergence from expert datasets across coverage, difficulty alignment, factual fidelity, and distractor quality. RefineLab pioneers a scalable, customizable path to reproducible dataset design, with broad implications for LLM evaluation. 1 Introduction Large language models (LLMs) have achieved remarkable success on a variety of natural language understanding and generation tasks, driving rapid progress in various areas ( 36 ; 11 ; 27 ) . To better evaluate and improve the capabilities of these models, central to this progress is the availability of high - quality datasets ( 31 ; 22 ; 35 ) . A high-quality evaluation dataset for LLMs is widely regarded as one that reliably reflects model capabilities, minimizes bias, and facilitates fair comparisons across models. Although this notion is broadly accepted within the community, there is no formal standard that explicitly defines what constitutes a high-quality dataset, or quantifies how the quality of a dataset influences evaluation reliability. In this work, we seek to reduce this ambiguity by identifying and analyzing key dimensions of dataset quality that can be systematically enhanced through careful sample refinement, while accounting for practical constraints such as token-level editing budgets. Figure 1 : An example of topic coverage (left) and difficulty level distribution (right) in the College Chemistry subset of MMLU before (Original) and after refinement (Refined). Targets follow MIT’s undergraduate chemistry major curriculum and feature a stress-testing difficulty profile. To ground our discussion, we take the widely adopted MMLU dataset ( 12 ) as an illustrative example. MMLU is often praised for its broad subject coverage and standardized multiple - choice format, which makes it suitable for evaluating knowledge recall and reasoning across diverse domains. However, despite its popularity, MMLU exhibits several key limitations: 1) Topic coverage: its empirical topic mix can diverge sharply from real-world evaluation standards, for example, in the College Chemistry subset shown in Figure 1 (left), biochemistry is completely absent, yet real - world college curricula, including MIT, Stanford, and other world-class universities, always demand a certain portion representation on it, resulting in datasets that fail to adequately reflect the full range of domain - specific skills and knowledge. 2) Imbalanced Difficulty Distribution: its fixed difficulty profile cannot accommodate varying evaluation goals, whether stress - testing advanced reasoning for math and science or supporting progressive skill diagnostics for language acquisition. As Figure 1 (right) shows, the original College Chemistry subset is dominated by medium-level difficulty questions that lack deep reasoning. Under an example of stress-testing target profiles, naive resampling can only remove excess medium items, drastically shrinking the dataset. Further refinement is needed to introduce new examples and maintain the effective dataset size while meeting specific difficulty requirements. 3) Factual and Logical Errors: its factual or logical mistakes that can persist unchecked, introducing misleading artifacts into model evaluation (for example, ( 31 ) pointed out that 6.5% of all questions in MMLU contain errors). Such errors can compromise the validity of benchmarking results. These shortcomings highlight the necessity of refining existing datasets to ensure reliable model evaluation and to drive progress in LLM development and downstream applications. While dataset refinement is essential, manual approaches are often tedious and inefficient. Recent work has introduced LLM-based methods for automatic dataset refinement, leveraging LLMs to synthesize new data with minimal human involvement ( 16 ; 21 ; 13 ; 8 ; 20 ) . Although these approaches are promising, several challenges remain: 1) Lack of explicit control: LLM-driven pipelines may unintentionally amplify coverage gaps or difficulty imbalances, as synthesized examples are often generated without fine-grained constraints. 2) Quality assurance limitations: Without rigorous validation, LLM-generated data can introduce new factual or logical errors, complicating overall quality control. 3) Resource allocation dilemma: Given that invoking high-capability models (e.g., GPT-4) for refinement incurs significant computational cost, efficiently allocating a limited budget of LLM calls across multiple refinement strategies to maximize dataset quality remains a challenging and largely unsolved problem. To end this, in this paper, we introduce RefineLab , a framework for textual dataset refinement that automatically selects and applies targeted editing operations to improve dataset quality under token-level budget constraints. In RefineLab, dataset high quality is operationalized via explicitly specified targets (e.g., desired topic distribution for coverage, difficulty levels, or factual accuracy). These targets can be defined either by users or inferred from real-world evaluation needs. Given such explicit requirements and a set of available refinement strategies, RefineLab employs an assignment module that selects the optimal refinement operations for each sample to maximize overall dataset quality, while respecting a global token-level budget constraint. Based on the estimated cost and quality gain of each refinement option, the assignment problem is formulated as an Integer Linear Programming (ILP) optimization, which determines, for each QA sample (q_{i},a_{i}) , which refinement operations ( r_{k} ) to apply: x_{ik} =1 if refinement operation r_{k} is assigned to sample (q_{i},a_{i}) , and x_{ik} = 0 otherwise. Overall, the objective is formalized as \addcontentsline{lla}{section}{\numberline q:ILP}\max_{x_{ik}\in\{0,1\}}\sum_{i,k}x_{ik}\cdot\Delta_{ik}\quad\text{s.t.}\quad\sum_{i,k}x_{ik}\cdot c_{ik}\leq B (1) where \Delta_{ik} denotes the estimated quality gain from applying r_{k} to sample (q_{i},a_{i}) , c_{ik} represents the token cost of that refinement, and B is the total available token budget. This formulation enables RefineLab to reason about trade-offs between different refinement operations and their associated costs, providing a principled way to enhance dataset quality under resource constraints. In this work, we make the following key contributions: • We propose RefineLab , the first LLM-driven framework for dataset refinement that formulates the refinement selection process as an integer linear program to optimize quality improvements under a token-level budget. • We introduce a set of modular refinement operations designed to improve QA datasets along a set of explicit quality targets, including topic coverage, difficulty calibration, and factual consistency. • We conduct comprehensive experiments on widely-used benchmarks such as MMLU, demonstrating that RefineLab produces higher-quality QA datasets with improved topic coverage, difficulty balance, and overall utility for evaluating LLM performance. 2 Related Work Synthetic

Read the complete text