Greedy Information Projection for LLM Data Selection
Victor Ye Dong, Kuan-Yun Lee, Jiamei Shuai, Shengfei Liu, Yi Liu, Jian Jiao · 2026
Read Greedy Information Projection for LLM Data Selection by Victor Ye Dong, Kuan-Yun Lee, Jiamei Shuai, Shengfei Liu, Yi Liu, Jian Jiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13790.txt.
Introduction
Dong and co-authors argue that good training data should cover the space of likely user queries. Their method, Greedy Information Projection, picks examples whose embeddings explain the largest left-over part of task scores. Scores can come from model judgments, metadata, or the shape of the data itself. The authors report that small picked subsets match or beat full-data fine-tuning. The same idea also works for pretraining data and for picking passages in retrieval tasks.
Catalog record
| CREATOR | Victor Ye Dong, Kuan-Yun Lee, Jiamei Shuai, Shengfei Liu, Yi Liu, Jian Jiao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74575 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13790
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Greedy Information Projection for LLM Data Selection Victor Ye Dong, Kuan-Yun Lee, Jiamei Shuai, Shengfei Liu, Yi Liu, Jian Jiao Abstract We present Greedy Information Projection ( GIP ), a principled framework for choosing training examples for large language model fine-tuning. GIP casts selection as maximizing mutual information between a subset of examples and task-specific query signals, which may originate from LLM quality judgments, metadata, or other sources. The framework involves optimizing a closed-form mutual information objective defined using both data and query embeddings, naturally balancing quality and diversity . Optimizing this score is equivalent to maximizing the projection of the query embedding matrix onto the span of the selected data, which provides a geometric explanation for the co-emergence of quality and diversity. Building on this view, we employ a fast greedy matching-pursuit procedure with efficient projection-based updates. On instruction-following and mathematical reasoning datasets, GIP selects small subsets that match full-data fine-tuning while using only a fraction of examples and compute, unifying quality-aware and diversity-aware selection for efficient fine-tuning. 1 Introduction A central question in the discussion of large language model (LLM) training is how one should select data. With recent developments of large-scale pretrained language models such as GPT-4 ( 1 ) , Qwen ( 46 ) and Llama ( 16 ) , the community has witnessed models achieving remarkable performance across a broad spectrum of natural-language tasks. Typically, these models train on large amounts of data that scale with model size, commonly known as the scaling law. Empirical scaling laws of LLMs reveal that model capability grows predictably with (i) parameter count, (ii) compute, and (iii) training data volume ( 18 ; 21 ) . Yet practitioners increasingly observe a subtler regime: once data quantity is sufficient, data quality becomes the primary factor limiting further gains. Consequently, a fundamental challenge in both pre-training and instruction tuning is to select a subset of samples that maximizes downstream performance while respecting resource constraints. Earlier work addressed the problem through diverse dataset construction ( 41 ; 40 ) and ad-hoc filtering heuristics such as perplexity thresholds, deduplication, or clustering ( 3 ; 48 ; 7 ; 15 ) . While effective, these heuristics offer little theoretical guidance on why a particular example is valuable, and they do not unify quality and diversity under a single objective. This work. We present a principled information-theoretic framework for data selection built on a mutual information (MI) metric between Gaussians scaled by data and query embeddings. As we will demonstrate later, this metric promotes both diversity and quality in a single objective, and also gives rise to efficient algorithm design. Our main contributions are: 1. Principled theoretical formulation. We propose a principled theoretical framework that casts the data selection problem as maximization of mutual information. Optimizing this objective naturally promotes diversity and quality of selected data. This framework is also flexible, enabling balanced selection with respect to disparate information sources, such as factuality and style scores from LLM evaluations. 2. Efficient approximation algorithms. We develop a greedy matching pursuit (MP) approximation algorithm that solves an approximate dual problem. The Greedy MP approach scales linearly with the total size of available data in practice, enabling data selection under realistic budget constraints. 3. Strong empirical results. On instruction-tuning benchmarks, our Greedy Information Projection ( GIP ) achieves substantial data efficiency gains over state-of-the-art baselines. Across MT-Bench, BBH, and GSM8K, GIP often approaches full-dataset performance using only 1–20% of training data, and in multiple settings matches or exceeds full-data results—corresponding to a substantial data reduction (see Figure 1 ). Figure 1: Left: GSM8K performance comparison on Qwen3-8B (left panel) and Mistral-7B (right panel) across different training data percentages (2.5%, 5%, 10%, 20%). Our proposed methods MP+MA and MP+SC are competitive with strong baselines (Random, DSIR, DISF, LESS), and often improve over Random/DISF/LESS. In several settings, they approach the full-dataset (100%, shown as a horizontal reference line) performance at 10%–20% of training data, demonstrating strong data efficiency. Right: Geometric interpretation of GIP . The method maximizes mutual information between Gaussian projections induced by the data embedding matrix F and score embedding matrix Q . This is equivalent to minimizing the volume (determinant) of score embeddings projected onto the null space of selected data, naturally balancing quality (high-score items) and diversity (new directions in embedding space). 2 Related Work 2.1 Data Curation for Large-Scale Language Models Diversity and quality are recognized as crucial factors impacting the overall quality of model training ( 34 ; 6 ; 5 ) . With the recent development of large language models, there is a growing trend in studies focusing on data composition quality and diversity, and how these factors influence language model pretraining and instruction tuning. For example, 27 highlight the importance of diversity in improving training efficiency and propose a method to remove near-duplicate records from training data. Similarly, 3 emphasize the significance of both quality and diversity in instruction tuning data, proposing a general method to balance these two aspects in training data selection. Furthermore, 11 introduce a systematic data selection approach that combines data quality, diversity, and augmentation for instruction tuning. However, neither ( 3 ) nor ( 11 ) have a unified framework to unify both quality and diversity; diversity was either applied as a penalty or considered in a different stage of data selection. Prior work has explored various data selection strategies. 7 propose instruction-following difficulty (IFD) scoring to identify high-quality instruction data. 15 introduce clustering-based active retrieval (CaR) that uses representativeness and uncertainty for data selection. 44 develop LESS, a gradient-based method for selecting informative training data. 45 propose data selection using importance resampling (DSIR) for domain adaptation. Our work differs by providing a unified information-theoretic framework that naturally balances quality and diversity in a single objective, while being computationally efficient and theoretically grounded. Optimal experimental designs for selecting diverse training examples are another strategy; see, for example ( 9 ) , for a discussion of how selecting diverse training examples relates to log-likelihood maximization. 2.2 Information-Theoretic Objectives in Selection and Clustering There is a long history of applications of mutual information to data selection and clustering. Mutual information can capture complex relationships between features that might not be apparent ( 22 ) . Mutual information is also flexible, working on various types of data such as categorical data ( 17 ) and numerical data ( 23 ) . Approximations of mutual information have also been applied to k-means ( 37 ; 4 ) . More recently, mutual information based algorithms have been developed for clustering ( 10 ) , community detection ( 32 ) , 3D object representation learning ( 33 ) , and unsupervised sentence embeddings ( 47 ) . Theoretical developments of mutual information inequalities have also been applied in pairwise comparisons ( 29 ) and generalized linear models ( 28 ) , where optimal selections and minimax algorithms are understood to be closely related to singular values of query matrices; see, e.g., ( 30 ) for a broad discussion. It is important to note that mutual information is often difficult to compute and various forms of estimators and approximations are used to reduce the computation cost; see ( 24 ) for a classic mutual