From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning
Yafu Li, Zhilin Wang, Tingchen Fu, Ganqu Cui, Sen Yang, Yu Cheng · 2025
Read From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning by Yafu Li, Zhilin Wang, Tingchen Fu, Ganqu Cui, Sen Yang, Yu Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11877.txt.
Introduction
Yafu Li and his five co-authors, publishing in 2025, teach language models a new trick: writing several draft answers and then merging them into one better answer. Their training method, Aggregation Fine-Tuning, shows the model many sets of drafts paired with a polished final response. At run time, the model drafts, merges, and can repeat the cycle to spend more computing for better results.
The gains are striking. A mid-sized open model trained this way on just 64,000 examples beat far larger models, including a 405-billion-parameter sibling and GPT-4, in judged head-to-head comparisons. The approach mixes two ways of spending test-time effort: drafting in parallel and refining in sequence. The authors pitch it as a route to stronger models without the usual answer of more data or more parameters.
Catalog record
| CREATOR | Yafu Li, Zhilin Wang, Tingchen Fu, Ganqu Cui, Sen Yang, Yu Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-11877
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning Yafu Li, Zhilin Wang, Tingchen Fu, Ganqu Cui, Sen Yang, Yu Cheng Abstract Scaling data and model size has been proven effective for boosting the performance of large language models. In addition to training-time scaling, recent studies have revealed that increasing test-time computational resources can further improve performance. In this work, we introduce Aggregation Fine-Tuning (AFT), a supervised fine-tuning paradigm where the model learns to synthesize multiple draft responses, referred to as proposals , into a single, refined answer, termed aggregation . At inference time, a propose-and-aggregate strategy further boosts performance by iteratively generating proposals and aggregating them. Empirical evaluations on benchmark datasets show that AFT-trained models substantially outperform standard SFT. Notably, an AFT model, fine-tuned from Llama3.1-8B-Base with only 64k data , achieves a 41.3% LC win rate on AlpacaEval 2, surpassing significantly larger LLMs such as Llama3.1-405B-Instruct and GPT-4. By combining sequential refinement and parallel sampling, the propose-and-aggregate framework scales inference-time computation in a flexible manner. Overall, These findings position AFT as a promising approach to unlocking additional capabilities of LLMs without resorting to increasing data volume or model size. Our code is publicly available at https://github.com/Linzwcs/AFT . † † affiliationnotice: Equal contribution 1 Introduction Large language models (LLMs) ( Dubey et al. 2024 ; OpenAI 2023 ) have demonstrated remarkable success in applications ranging from coding ( Li et al. 2023a ; Roziere et al. 2023 ) and reasoning ( Yuan et al. 2023 ; Qin et al. 2024a ) to AI assistance ( Chiang et al. 2023 ) , driven by the expansion of both data and model capacity during pre-training ( Hoffmann et al. 2022 ; Kaplan et al. 2020 ) and supervised fine-tuning (SFT) ( Yuan et al. 2023 ; Zhang et al. 2024a ) . Beyond training-time scaling, recent studies ( Snell et al. 2024a ; Zhang et al. 2024c ; Wang et al. 2024a ) indicate that allocating additional compute at inference can further boost performance, enabling smaller models to rival larger ones. Correspondingly, a variety of inference-time scaling algorithms ( Madaan et al. 2023 ; Brown et al. 2024 ; Świechowski et al. 2023 ) have emerged, including sequentially revising generations or selecting among parallel samples. Figure 1 : Aggregation Fine-Tuning: Learning to generate a refined response by synthesizing multiple drafts alongside the query, different from the direct query-response mapping of traditional supervised fine-tuning. Despite the growing complexity of inference algorithms, typical SFT pipelines still learn a direct mapping from a user query to a reference response. To this end, a line of work attempts to integrate the intermediate inference process such as rationale ( Zelikman et al. 2022 ) and reasoning traces ( Tian et al. 2024 ; Zhang et al. 2024b ) , into supervised training to improve test-time performance. While these intermediate steps have been shown effectiveness in reasoning tasks such as coding and math, they are less straightforward to extend to tasks such as writing or open-ended dialogue. The intermediate process for general tasks remains less discussed. In this work, we seek to find an inference process for instruction-following and integrate it into supervised training to facilitate test-time scaling. Recently, Wang et al. 2024a propose mixture-of-agents (MoA) which aggregates diverse response from multiple agents to improve generation quality. Drawing inspiration from the mixture-of-agents (MoA) framework ( Wang et al. 2024a ) , which aggregates diverse outputs from multiple LLMs, we propose to treat aggregation itself as a general-purpose inference process for instruction following. This leads to a new supervised training framework, which we term aggregation fine-tuning ( AFT ). As illustrated in Figure 1 , under AFT the model is trained to aggregate multiple draft responses into a single, refined answer rather than directly mapping a query to a response. During inference, the model can itself generate and aggregate draft responses to produce a refined response. To perform aggregation fine-tuning, we begin by constructing a specialized training dataset in which each instance comprises a query, multiple draft responses (i.e., proposals ), and a refined response (i.e., aggregation ). We consider two types of proposals: off-policy and on-policy proposals, differing on whether or not the proposals are sampled from the model to be trained. Subsequently, the model is trained to aggregate proposals to an aggregation given the query, thereby conventional fine-tuning of query-to-response alignment. During inference, the model can perform propose-and-aggregate in a multi-layer fashion, which can be regarded as MoA ( Wang et al. 2024a ) with only one agent. At each layer, the model aggregates its proposals at the previous layer and generate proposals for the next layer. This iterative process of proposal generation and aggregation allows the model to leverage both parallel sampling and sequential revision effectively. Empirical results on Llama3.1-8B-Base and Mistral-7B-v0.1-Base demonstrate that AFT substantially surpasses standard SFT in various benchmark datasets. On AlpacaEval 2, AFT models improve LC win rates by up to 13.9% on average, with on-policy AFT models exhibiting particularly significant improvements. Further performance gains are achieved through propose-and-aggregate at inference time, elevating Mistral-based models above a 30% LC win rate and Llama-based models beyond 40%. Notably, the top-performing model, fine-tuned on only 64k training examples without preference alignment , attains an LC win rate of 41.3% , surpassing strong competitors such as Llama3.1-405B-Instruct and GPT4. In MT-Bench, propose-and-aggregate pushed the best-performing mode to a score of 8.1. Additionally, AFT shows enhancements over SFT in downstream tasks such as GSM8K and IFEval. Analytic findings reveal that by fitting answers conditioned on drafts, aggregation learning benefits from shaping a low-perplexity region, where the model converges in a faster and more stable way. We also observe that both the diversity and quality of proposals contribute to the aggregation quality. In addition, the propose-and-aggregate method effectively combines sequential refinement and parallel sampling to scale inference-time computation, producing consistent improvements across multiple tasks with manageable computational overhead. Overall, these insights highlight aggregation learning and propose-and-aggregate inference as a flexible, efficient strategy for unlocking the full potential of LLMs. We will release the code, data and model after the anonymous period. 2 Related Work Supervised Fine-tuning of LLMs. Supervised fine-tuning plays a pivotal role in enhancing the instruction-following ability of LLM ( Li et al. 2023b ; Zheng et al. 2023 ; Liu et al. 2024b ; Ouyang et al. 2022a ; Qin et al. 2024b ; Jiang et al. 2024 ; Zhou et al. 2023b ) . Previous works find that high-quality instruction-following data is essential for the success of SFT ( Zhao et al. 2024 ; Liu et al. 2024a ; Xu et al. 2024 ; Zhou et al. 2023a ; Li et al. 2024a ) and various data-curation techniques are proposed ( Wang et al. 2023 ; Du et al. 2023 ; Chen et al. 2024a ; Li et al. 2024b ; Sun et al. 2023 ) . More specifically, including complex constrained ( Sun et al. 2024 ; He et al. 2024 ) , multi-faced ( Lou et al. 2024 ) , self-refined ( Zelikman et al. 2022 ; Cheng et al. 2024 ; Wu et al. 2024a ) and sequential ( Hu et al. 2024 ) data could enhance the