Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement
Maosong Cao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Haodong Duan, Songyang Zhang, Kai Chen · 2025
Read Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement by Maosong Cao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Haodong Duan, Songyang Zhang, Kai Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12273.txt.
Introduction
Maosong Cao and colleagues generate synthetic conversation data to fine-tune chat models, using stored world knowledge and self-critique to keep quality high. Human-written fine-tuning data is scarce and expensive, yet model manners depend on it. Writing in twenty twenty-five, their two-stage framework first grows a tree of world knowledge topics to seed diverse questions. Answers, then runs a self-reflection stage where the model revises its own drafts. The pipeline is designed to scale without armies of annotators.
A base model trained on only a modest set of the synthetic examples outperformed counterparts trained otherwise, by the authors report. The refinement stage also let models of many sizes, up to the largest tested, improve themselves iteratively. A side study of how performance scales with synthetic data volume suggests large headroom remains.
Catalog record
| CREATOR | Maosong Cao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Haodong Duan, Songyang Zhang, Kai Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement Maosong Cao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Haodong Duan, Songyang Zhang, Kai Chen Abstract The quality of Supervised Fine-Tuning (SFT) data plays a critical role in enhancing the conversational capabilities of Large Language Models (LLMs). However, as LLMs become more advanced, the availability of high-quality human-annotated SFT data has become a significant bottleneck, necessitating a greater reliance on synthetic training data. In this work, we introduce Condor , a novel two-stage synthetic data generation framework that incorporates World Knowledge Tree and Self-Reflection Refinement to produce high-quality SFT data at scale. Our experimental results demonstrate that a base model fine-tuned on only 20K Condor-generated samples achieves superior performance compared to counterparts. The additional refinement stage in Condor further enables iterative self-improvement for LLMs at various scales (up to 72B), validating the effectiveness of our approach. Furthermore, our investigation into the scaling for synthetic data in post-training reveals substantial unexplored potential for performance improvements, opening promising avenues for future research. 1 1 1 This work is done when Taolin Zhang and Chuyu Zhang are on internship at Shanghai AI Laboratory, ∗ * means equal contribution, † \dagger means corresponding author, ‡ \ddagger means project lead. 1 Introduction Large Language Models (LLMs) are experiencing rapid advancements, with proprietary models such as like GPT ( Achiam et al., 2023 ) and Gemini ( Team et al., 2023 ) , alongside open-source counterparts such as LLaMA ( Dubey et al., 2024 ) , Qwen ( Yang et al., 2024 ) , Mistral ( Jiang et al., 2023a ) , and Deepseek ( Liu et al., 2024a ) evolving at an unprecedented pace. However, this rapid iteration comes with a significant challenge: the depletion of existing high-quality data for Supervised Fine-Tuning (SFT). Moreover, the internet is increasingly inundated with synthetic data of varying and often questionable quality, making it ever more difficult to construct and filter higher-quality training data for LLMs. Figure 1: Comparison between Official Models and Condor-Refined Models. The performances are obtained from CompassJudger-1-32B. Different families of popular LLMs benefit greatly from the Condor Refine engine. Empirical studies on SFT training have conclusively shown that both data quality and quantity play crucial roles in enhancing model performance ( Shen, 2024 ) . While scaling laws suggest that models can achieve extraordinary capabilities when trained on large datasets, high-quality data can yield comparable results even at smaller scales. This highlights the importance of generating substantial amounts of high-quality data to further enhance large-scale model capabilities. Recent research introduce various methods for synthesizing SFT training data ( Wang et al., 2024a ; Ding et al., 2023 ; Xu et al., 2023 ; Yuan et al., 2024 ; Tang et al., 2024 ) . However, these approaches face several limitations: they often rely heavily on existing datasets for seed prompts, lack prompt diversity, and cannot guarantee the quality of the generated data. To address these challenges, we introduce Condor , an automated two-stage pipeline for synthetic data generation that effectively leverages existing world knowledge. Our approach begins with constructing a comprehensive World Knowledge Tree by prompting LLM to generate tags, and then creating a complete tag chain from coarse to fine through an auto-regressive pipeline. After generating diverse questions and original responses using tags, we then employ Self-Reflection Refinement to obtain synthesized SFT data with more higher-quality replies. Condor further generates extensive and high-quality SFT data by integrating up-to-date world knowledge tags. In this way, Condor not only ensures the relevance and diversity of the content but also maintains scalability in data synthesis. We conduct experiments on multiple models using the data synthesized by Condor, as shown in Figure 1 , the results indicate that the data synthesized by Condor significantly improves the subjective chat capabilities of the models across the board. To summarize, our contributions are as follows: \bullet We propose Condor Void , a novel approach incorporating a knowledge inspiration strategy based on the World Knowledge Tree, enabling models to acquire rich world knowledge for generating diverse questions. \bullet We further introduce Condor Refine , a self-reflection mechanism that allows models to iteratively refine their responses during the construction phase, resulting in higher-quality responses. \bullet We investigate the scaling and self-iteration techniques for Condor datasets, offering valuable insights and potential pathways for the scalable expansion of synthetic data. 2 Related Work 2.1 Synthesizing Instruction Tuning Data With the rapid advancement of Large Language Models (LLMs) ( Achiam et al., 2023 ; Team et al., 2023 ; Dubey et al., 2024 ; Yang et al., 2024 ; Jiang et al., 2023a ; Liu et al., 2024a ) and Large Multi-Modality Models (LMMs) ( Radford et al., 2021 ; Zhu et al., 2023 ; Zhang et al., 2024a ; Zhang et al., 2024b ; Liu et al., 2024b ; Chen et al., 2024 ) , the demand for high-quality training data continues to grow. As a result, synthetic data generation has become increasingly crucial for model development. While high-quality synthetic data can significantly enhance model performance, low-quality synthetic data may lead to model degradation or collapse. WizardLM ( Xu et al., 2023 ) introduces an Evol-Instruct scheme, which expands an initial question dataset by generating new questions through techniques such as deep thinking, adding constraints, and reconstruction. However, the quality and diversity of the generated data are strongly influenced by the characteristics of the original dataset. Similarly, Self-Reward ( Yuan et al., 2024 ) builds on a seed dataset, enabling the model to generate questions via few-shot prompting and produce multiple corresponding responses. These responses are subsequently evaluated by a reward model and labeled as training data for reinforcement learning with human feedback (RLHF). Magpie ( Xu et al., 2024 ) explores leveraging models themselves to generate synthetic data by utilizing chat templates as prompts to directly produce dialogues. However, this approach is not universally applicable to all model types, and therefore lacks generalization ability. Additionally, the synthetic data generated often lacks stability, containing significant noise that requiring extensive filtering to maintain quality. Agent-based data synthesis methods, such as MATRIX-Gen ( Tang et al., 2024 ) , simulate multiple world scenarios using a multi-agent framework and collect dialogue data from interactions within these scenarios. While effective in some cases, this method is time-consuming, labor-intensive, and limited by the need for multiple models to operate concurrently. In contrast to these methods, Condor eliminates the dependence on a seed dataset, requiring only tags as seeds for expansion and generation. This enables exceptional diversity and scalability in data production. Furthermore, Condor handles all data generation tasks with a single model, significantly reducing costs while maintaining high efficiency. Figure 2: The Pipeline of Condor Engine. The pipeline is mainly divided into (1) Data synthesis: LLM is inspired from the World Knowledge Tree with different task diversity and difficulty to construct the original question-answer dataset \boldsymbol{{D}_{V}} ; (2) Data refinement: LLM is utilized to reflect on the synthesized data and generate critiques for refinement and improve the quality of responses, producing the dataset \boldsymbol{{D}_{R}} . 2.2 LLM Self-Iteration Recent studies have explored methods for large language models (LLMs) to improve themselves through iterative self-enhancement. For instance, I-SHEEP ( Liang