CrowdSelect : Synthetic Instruction Data Selection with Multi-LLM Wisdom
Yisen Li, Lingfeng Yang, Wenxuan Shen, Pan Zhou, Yao Wan, Weiwei Lin, Dongping Chen · 2025
Read CrowdSelect : Synthetic Instruction Data Selection with Multi-LLM Wisdom by Yisen Li, Lingfeng Yang, Wenxuan Shen, Pan Zhou, Yao Wan, Weiwei Lin, Dongping Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01836.txt.
Introduction
Small language models learn to follow instructions by training on examples distilled from big ones, and the examples chosen matter as much as the volume. Li and co-authors argue that judging examples by one signal, a reward score or a perplexity number, misses most of what makes an example good.
Their method, CrowdSelect, builds three base measures from the wisdom of several large models at once: how diverse their answers are, and how reward models grade the results. A clustering step then keeps the selected set varied across topics. Across four base models, the measures improved scores on two hard chat benchmarks. The full method set the best marks in both complete and light fine-tuning. For one small model it gained 4.81 percent on Arena-Hard and 11.1 percent on MT-Bench.
Catalog record
| CREATOR | Yisen Li, Lingfeng Yang, Wenxuan Shen, Pan Zhou, Yao Wan, Weiwei Lin, Dongping Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01836
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CrowdSelect : Synthetic Instruction Data Selection with Multi-LLM Wisdom Yisen Li, Lingfeng Yang, Wenxuan Shen, Pan Zhou, Yao Wan, Weiwei Lin, Dongping Chen Abstract Distilling advanced Large Language Models’ instruction-following capabilities into smaller models using a selected subset has become a mainstream approach in model training. While existing synthetic instruction data selection strategies rely mainly on single-dimensional signals ( i.e. , reward scores, model perplexity), they fail to capture the complexity of instruction-following across diverse fields. Therefore, we investigate more diverse signals to capture comprehensive instruction-response pair characteristics and propose three foundational metrics that leverage Multi-LLM wisdom, informed by (1) diverse LLM responses and (2) reward model assessment. Building upon base metrics, we propose CrowdSelect , an integrated metric incorporating a clustering-based approach to maintain response diversity. Our comprehensive experiments demonstrate that our foundation metrics consistently improve performance across 4 base models on MT-bench and Arena-Hard. CrowdSelect , efficiently incorporating all metrics, achieves state-of-the-art performance in both Full and LoRA fine-tuning, showing improvements of 4.81% on Arena-Hard and 11.1% on MT-bench with Llama-3.2-3b-instruct. We hope our findings will bring valuable insights for future research in this direction. Code are available at https://github.com/listentm/crowdselect . † † † Equal Contribution. \ddagger Project Leader. † † * Corresponding Authors. 1 Introduction In recent years, Large Language Models (LLMs) ( Achiam et al. 2023 ; Jaech et al. 2024 ; Team et al. 2024 ; Guo et al. 2025 ) have demonstrated remarkable capability in following user instructions to generate coherent and contextually helpful responses ( Jiang et al. 2023 ; Zheng et al. 2023b ; Wen et al. 2024 ) . Yet, the computational overhead for instruction tuning and massive parameter sizes of these models create a considerable barrier to practical deployment ( Peng et al. 2023 ) . To address this, many approaches distill the instruction-following ability of advanced LLMs into smaller, more efficient models through a small-scale instruction tuning process with synthetic responses ( Xia et al. 2024 ; Zhou et al. 2024a ) . A critical bottleneck, however, lies in selecting the optimal data for this distillation process. Most existing data selection methods rely on predefined rules ( Chen et al. 2023a ) , automated single-dimensional signals — such as reward scores ( Wu et al. 2024b ; Lambert et al. 2024 ) or difficulty metrics ( Li et al. 2023b ; Li et al. 2024b ) — to identify valuable examples for fine-tuning. While effective to some extent, such narrow signals may overlook essential nuances of user instructions, especially when instructions contain challenges from diverse fields ( Händler 2023 ; Feng et al. 2025 ) . This raises a fundamental question: “Can we leverage multi-dimensional signals to better reflect the various facets of each sample for more effective instruction tuning data selection?” Figure 1: A demonstration of instruction tuning with selected synthetic instruction-response pairs. Inspired by previous works that leverage Multi-LLM collaboration ( Guo et al. 2024 ; Lu et al. 2024 ) , we take an explorative step towards more robust and comprehensive data selection by introducing CrowdSelect , a framework that treats pre-collected Multiple LLMs’ responses and their reward scores as different reflections of the instruction to leverage Multi-LLM Wisdom. Instead of treating each instruction–response pair in isolation — typically with just a single model — our method aggregates multiple responses for each instruction from a diverse set of LLMs. Crucially, we also factor in each response’s score provided by various reward models. This multi-view setup captures more “facets” of each instruction, illuminating subtle differences in how various models handle the same query. Based on these observations, we propose three base explorative metrics: • Difficulty - Identifies instructions on which the majority of models struggle, surfacing challenging prompts critical to learning. • Separability – Highlights instructions whose response quality exhibits high variance across models, making them especially useful for differentiating stronger from weaker capabilities. • Stability – Measures how consistently model performance follows expected size-based ranking across families, ensuring the selected data helps reinforce well-grounded alignment signals. Our exploratory experiments in full fine-tuning (FFT) and low-rank adaptation (LoRA) ( Hu et al. 2021 ) experiments on LLama-3.2-3b-base/instruct ( Dubey et al. 2024 ) and Qwen-2.5-3b-base/instruct ( Yang et al. 2024b ) demonstrate the robustness and efficacy of our proposed metrics through significant performance gaps between top-scored and bottom-scored data subset fine-tuning, with potential further improvements through metric combination. Subsequently, we propose CrowdSelect that combines these metrics with a clustering strategy to preserve diversity and explore the upper bound of leveraging Multi-LLM wisdom to identify a compact yet high-impact subset of instruction-response data. Experimental results show that models fine-tuned on our selected subset significantly outperform baselines and previous state-of-the-art data selection methods, achieving improvements of 4.81% on Arena-Hard and 11.1% on MT-bench with Llama-3.2-3b-instruct. Furthermore, CrowdSelect achieves state-of-the-art performance across four models on two benchmarks, demonstrating both the generalizability and robustness of our selected data and methodology, paving a new dimension for efficient instruction tuning. Our contributions are summarized as follows: • Investigation of Multi-LLM Wisdom in Instruction Data Selection. We propose a novel approach that utilizes multiple synthesized responses from different LLMs for each instruction, enhancing the diversity and quality of data. • Novel Metrics and Methods. We design three new explorative base metrics— Difficulty , Separability , and Stability —that leverage multi-LLM responses and reward scores as more comprehensive signals, and combine them into CrowdSelect to explore the upper bound in selecting high-quality data for instruction tuning. • State-of-the-art Performance. We demonstrate that combining our metrics and clustering techniques for data selection leads to a new SOTA in efficient instruction tuning in both Llama-3.2-3b and Qwen-2.5-3b. 2 Related Work Instruction Tuning Data Selection. Instruction Tuning stands out to be a method to solve the gap between pre-trained knowledge and real-world user scenarios ( Ouyang et al. 2022 ; Bai et al. 2022 ) . Recent efforts like Vicuna ( Peng et al. 2023 ) and LIMA ( Zhou et al. 2024a ) demonstrate high performance with a carefully selected small dataset, highlighting the growing importance of efficient instruction tuning. Three key metrics determine instruction data quality: Difficulty , Quality , and Diversity . Difficulty , focusing mainly on the question side, is considered more valuable for model learning ( Li et al. 2023b ; Li et al. 2024b ; Liu et al. 2024b ; Lee et al. 2024 ; Wang et al. 2024b ) . Quality , mainly addressing the response side, measures the helpfulness and safety of model responses, typically assessed using LLM evaluators ( Chen et al. 2023a ; Chen et al. 2024b ; Liu et al. 2024c ; Ye et al. 2024 ) , reward models ( Son et al. 2024 ; Lambert et al. 2024 ) , and gradient similarity search ( Xia et al. 2024 ) . Diversity also plays a crucial role in covering various instruction formats and world knowledge, primarily improving model robustness ( Bukharin and Zhao 2023 ; Wang et al. 2024d ) . Data Synthesis for Instruction Tuning. While the development of LLMs initially relied on human-curated instruction datasets for instruction tuning ( Zheng et al. 2023a ; Zhao et al. 2024 ; Lightman et al. 2023