DUET: Optimizing LLM Training Data Mixtures via Noisy Feedback from Unseen Evaluation Tasks
Zhiliang Chen, Gregory Kang Ruey Lau, Chuan-Sheng Foo & Bryan Kian Hsiang Low · 2025
Read DUET: Optimizing LLM Training Data Mixtures via Noisy Feedback from Unseen Evaluation Tasks by Zhiliang Chen, Gregory Kang Ruey Lau, Chuan-Sheng Foo & Bryan Kian Hsiang Low (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00270.txt.
Introduction
Zhiliang Chen and co-authors tuned training data mixes for unseen tests in a 2025 paper. A language model's skill hinges on how well its training data fits the task it will face. Often that task is hidden, as when user chats are encrypted end to end, so the right mix cannot be read off in advance.
Their method, DUET, learns the mix from noisy feedback instead. The model is fielded on the hidden task over many rounds, and scores such as user ratings trickle back. Those scores steer the next mix toward data that helps. The loop turns blind deployment into a guide for better fine-tuning data.
Catalog record
| CREATOR | Zhiliang Chen, Gregory Kang Ruey Lau, Chuan-Sheng Foo & Bryan Kian Hsiang Low |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-00270
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DUET: Optimizing LLM Training Data Mixtures via Noisy Feedback from Unseen Evaluation Tasks Zhiliang Chen, Gregory Kang Ruey Lau, Chuan-Sheng Foo & Bryan Kian Hsiang Low Abstract The performance of an LLM depends heavily on how well the training data matches the downstream evaluation task. However, in many practical settings, we typically do not know the data in the evaluation task (e.g., conversations between a chatbot and users are end-to-end encrypted). We refer to such tasks as unseen evaluation tasks . We can only deploy the LLM on these unseen evaluation tasks to gather multiple rounds of feedback on how well the model performs (e.g., gathering user ratings from a chatbot). In addition, this feedback can be noisy. How can we exploit such noisy feedback efficiently to optimize the LLM training data-mixture? Our paper presents DUET , a novel global-to-local algorithm that optimizes training data mixtures by interleaving data selection with Bayesian optimization to exploit coarse and noisy feedback from a downstream evaluation task. DUET is flexible enough to incorporate different data selection methods, each with different performance-compute tradeoffs. By analyzing DUET’s cumulative regret , we theoretically show that DUET converges to the optimal training data mixture even without any fine-grained data information from an unseen task. Finally, our experiments across a variety of language tasks demonstrate that DUET attains substantial performance improvements over existing data selection and mixing methods in the unseen-task setting. Our library, which is flexible enough to optimize different LLM training ingredients, can be found at https://github.com/chenzhiliang94/BO-for-LLMs . 1 Introduction The performance of an LLM depends heavily on how well the training data domain ( Chen et al., 2024a ; Xie et al., 2023a ) matches the downstream evaluation task ( Hoffmann et al., 2022 ; Long et al., 2017 ) . For instance, if we knew that LLM users are interested in asking layman science questions, then training or fine-tuning the LLM with more Wikipedia data allows it to converse better with these users. Hence, knowing the evaluation task is important for curating a more relevant training data mixture, producing an LLM with better performance over the specific task of interest. Unfortunately, in practice, the data (e.g., its domain, distribution, or labels) involved in an unseen evaluation task are often unknown. Thus, it is not obvious what data is relevant for training or fine-tuning the model. Consider the following problem setting: An LLM owner is interested in fine-tuning their LLM to converse better with users but due to privacy concerns ( Li et al., 2024 ) , conversations between the deployed LLM and users are end-to-end encrypted ( openai.com/enterprise-privacy ). Hence, the LLM owner does not know the actual evaluation data seen during test-time. Rather, they only receive coarse, noisy feedback on how well the LLM has performed in the conversation (e.g., user ratings or duration spent on the application). Of course, a naive idea is to simply iterate through all possible data mixtures and observe the resulting LLM performance, which is too computationally expensive. A better question is to perhaps ask: How can we exploit the noisy feedback loop efficiently to improve and optimize the training data mixture? This paper presents DUET (Fig. 1 ), an efficient algorithm that exploits a noisy feedback loop to optimize the training D ata mixture for an U nseen E valuation T ask. DUET is a global-to-local algorithm that interleaves data selection ( Albalak et al., 2024 ; Ting and Brochu, 2017 ; Koh and Liang, 2017 ) with Bayesian optimization (BO) ( Snoek et al., 2012 ; Srinivas et al., 2010 ) to optimize the training data mixture. Globally, BO in DUET uses coarse, noisy feedback from the unseen evaluation task to automatically refine the mixing ratio of data domains in the training data mixture iteratively. Locally, DUET uses data selection to retrieve high-quality data points from each data domain until the proposed mixing ratio is reached. This results in an algorithm that can optimize training data iteratively even without having access to fine-grained data information from the evaluation task. Figure 1: DUET exploits a feedback loop to optimize the data mixture for an unseen evaluation task. In contrast, conventional data mixing and selection works require fine-grained data information of the task, which is not available here. Related works. In our problem setting, (a) there is no direct access to the data (e.g., its domain, distribution, or labels) involved in the unseen evaluation task but (b) we can gather multiple rounds of feedback (details covered in Sec. 2.2 ) from the task using an LLM. App. A.1 provides a few more practical examples of this setting. This setting is different from those considered in conventional domain adaptation (DA) and domain generalization (DG) works. Prior DA works assume fine-grained knowledge of data (e.g., labeled/unlabeled data ( Zhang et al., 2022 ) or data distribution ( Ganin and Lempitsky, 2015 ; Zhang et al., 2021 ) ) from the evaluation task for selecting relevant training data that match the evaluation data. On the other hand, DG considers a rigid setting with no knowledge (not even feedback) of the evaluation task ( Muandet et al., 2013 ; Shin et al., 2024 ; Wang et al., 2022 ) . Similarly, data mixing works such as DoReMi ( Xie et al., 2023a ) , BiMix ( Ge et al., 2025 ) and more ( Chen et al., 2024a ; Fan et al., 2024 ; Xie et al., 2025 ; Xie et al., 2023b ) introduced methods to optimize data mixtures, and data selection works ( Albalak et al., 2024 ; Xia et al., 2024 ; Pruthi et al., 2020 ; Xie et al., 2023b ) explored ways to find high-quality data to improve an LLM’s performance. However, these methods assume some availability of fine-grained evaluation data information, such as evaluation gradients, labels, distribution or naively assuming the training data shares the same distribution as the task. In practice (like in our setting), these are not always available. In fact, when we applied existing data mixing and selection methods directly to our setting, they perform worse than DUET (Sec. 6 ). We provide more discussion of the shortfalls of these prior works in App. A.2 . To the best of our knowledge, DUET is the first work that interleaves data selection with BO to iteratively optimize training data mixture based on feedback from an unseen evaluation task. At first glance, eliciting multiple rounds of feedback with BO seems expensive. However, BO is sample-efficient and is the only way we can exploit such coarse and noisy feedback iteratively, unlike prior methods that require much more fine-grained data information (see above). In fact, subjecting models to multiple rounds of training or fine-tuning in a feedback loop is a natural part of the deployment life-cycle to improve LLMs. Specifically, our contributions are: \bullet We introduce a novel and realistic problem setting where the data involved in an unseen evaluation task is unknown but we can deploy our LLM to gather multiple rounds of coarse and noisy feedback. Then, we introduce DUET , a novel algorithm that exploits the feedback loop to optimize training D ata mixture for the U nseen E valuation T ask. To achieve