LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points
Xuemiao Zhang, Can Ren, Chengying Tu, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai · 2025
Read LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points by Xuemiao Zhang, Can Ren, Chengying Tu, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01317.txt.
Introduction
Zhang and six co-authors built a giant synthetic question-answer dataset for training language models in twenty twenty-five. Good training data is scarce, so they generated their own. Their framework, LinkSyn, starts from seed questions, extracts the knowledge points each one tests, and links seeds sharing points into a graph.
Walks through the graph gather tightly related seeds, which a strong reasoning model then fuses into new, harder questions. A scoring function balances covering rare knowledge points against popular ones. The result, LinkQA, holds fifty billion tokens across many subjects. Continual pre-training of an eight-billion-parameter model on LinkQA raised scores on two major knowledge benchmarks by about eleven and a half percent on average. Gains held across model sizes and training scales, the authors report.
Catalog record
| CREATOR | Xuemiao Zhang, Can Ren, Chengying Tu, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 98 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01317
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points Xuemiao Zhang, Can Ren, Chengying Tu, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai Abstract The advancement of large language models (LLMs) struggles with the scarcity of high-quality, diverse training data. To address this limitation, we propose LinkSyn, a novel knowledge point (KP) graph-based synthesis framework that enables flexible control over discipline and difficulty distributions while balancing KP coverage and popularity. LinkSyn extracts KPs from question-answering (QA) seed data and constructs a KP graph to synthesize diverse QA data from multiple seeds strongly linked by KPs and sampled from graph walks. Specifically, LinkSyn incorporates (1) a knowledge distribution value function to guide the adjustment of path sampling probability and balance KP coverage and popularity during graph walks; (2) diffusion-based synthesis via DeepSeek-R1 by leveraging multiple seeds with dense logical associations along each path; and (3) high-difficulty QA enhancement within given disciplines by flexible difficulty adjustments. By executing LinkSyn, we synthesize LinkQA, a diverse multi-disciplinary QA dataset with 50B tokens. Extensive experiments on Llama-3 8B demonstrate that continual pre-training with LinkQA yields an average improvement of 11.51% on MMLU and CMMLU, establishing new SOTA results. LinkQA consistently enhances performance across model size and initial FLOPs scales. 1 1 1 The core code and dataset will be made available at link. 1 Introduction As the scale of large language models (LLMs) escalates exponentially, the scarcity of high-quality training data has emerged as a critical bottleneck ( Muennighoff et al. 2025 ; Villalobos et al. 2024 ) , particularly in multi-disciplinary domains ( Kandpal et al. 2023 ) . Data synthesis has consequently gained prominence as a viable solution, offering scalable production of domain-specific knowledge representations ( Gunasekar et al. 2023 ; Li et al. 2023 ; Nadăș, Dioșan, and Tomescu 2025 ) . Crucially, synthetic data in question–answer (QA) format has demonstrated significant efficacy in enhancing model performance on knowledge-intensive tasks by providing structured reasoning pathways and explicit knowledge representations ( Chen et al. 2025 ; Wang et al. 2025 ; Maini et al. 2024 ) . Despite these advances, current synthesis methods that depend on seed corpora face significant limitations. First, single-seed synthesis using trained models often experiences limited diversity due to inherent model biases ( Qin et al. 2025 ; Su et al. 2025 ; Akter et al. 2025 ; Zhou et al. 2025 ) . Second, entity-based methods ( Qin et al. 2025 ; Jiang et al. 2025b ; Yang et al. 2025 ) , which extract sets of entities mentioned in documents and link documents through entity co-occurrence, aim to synthesize data from multiple connected documents. However, these methods exhibit limited knowledge integration, as individual entities rarely represent the document’s core subject. Consequently, such connections lack semantic coherence, thus restricting cross-textual knowledge integration. Additionally, current methods struggle to finely adjust the distributions of synthesized data in terms of difficulty, discipline, and knowledge popularity. This results in a low yield of valuable data and poor performance on benchmarks that require higher-order abilities, such as reasoning ( Hendrycks et al. 2021a ) . Figure 1: Comparison between LinkQA and baselines. Intuitively, unlike documents that mention numerous entities, a QA instance typically examines a few knowledge points (KPs), allowing each KP to serve as a strong representation of the QA itself. Thus, KP co-occurrence inherently provides more logical connections between QAs. Building on this insight, we propose constructing a KP graph from QA seeds to capture robust logical associations. We then introduce LinkSyn, a novel diversity-driven and theoretically rigorous synthesis framework that traverses this graph to generate multi-seed QAs with dense logical links. Specifically, LinkSyn: (1) introduces knowledge distribution values to guide the adjustment of path sampling probability, balancing KP coverage and popularity during graph traversal; (2) synthesizes diverse or entirely novel QAs via DeepSeek-R1 ( DeepSeek-AI et al. 2025 ) in a diffusion-based manner by leveraging multiple seeds with dense logical associations along each path; and (3) enhances the concentration of high‑difficulty QAs within specified disciplines during synthesis by flexibly adjusting the difficulty levels and discipline proportions of the sampling seed instances along graph paths. We conduct extensive experiments by continually pre-training Llama-3 8B ( Dubey et al. 2024 ) at the 2T-token checkpoint using LinkQA, a multi-disciplinary QA dataset synthesized by LinkSyn. As shown in Figure 1 , LinkQA achieves an average improvement of 11.51% over the pre‑training baseline on MMLU and CMMLU, attaining state‑of‑the‑art (SOTA) results. LinkQA also demonstrates scalable performance gains as model sizes expand from 1.7B to 16B and checkpoints progress from 2T-token to 10T-token. Our main contributions are summarized as follows: • We propose constructing a KP graph based on KPs extracted from seed QA data, and introduce a theoretically rigorous framework, LinkSyn, to synthesize diverse QAs from multiple seeds that are strongly linked by KPs. • We dedicate substantial resources to executing LinkSyn to synthesize LinkQA, a diverse multi-disciplinary QA dataset with 50B tokens. LinkQA is controllable in terms of difficulty, discipline, and KP distributions, fostering community research in scalable data synthesis and LLM advancement. • Extensive experiments conducted on Llama-3 8B trained with 40B tokens demonstrate that our LinkQA improves by an average of 11.51% on MMLU and CMMLU and achieves SOTA average performance on 12 benchmarks. 2 Method Figure 2: An overview of the LinkSyn pipeline. Step 1: Knowledge Point Graph Construction — We construct the KP graph based on the co-occurrence of KPs in the seed data. Step 2: Knowledge Value-Guided Path Sampling — Two sampling policies are utilized to balance KP coverage and popularity. Step 3: Attribute-Guided Instance Sampling — Instances are sampled by controlling the distributions of difficulty and discipline. 2.1 Overview LinkSyn is a framework designed to combine multiple seed QA instances to generate diverse samples that conform to expected distributions. As illustrated in Figure 2 , we first extract representative KPs from each QA instance and construct a KP graph where the edges denote strong logical relationships based on KP co-occurrence patterns. We then navigate this knowledge space using two complementary graph walking policies: popularity priority, which favors central KPs, and coverage priority, which explores diverse regions of the graph. For each KP along the generated paths, we sample seed instances according to specified difficulty levels and discipline distributions. Our diffusion-based approach then combines logically related instances to create novel QA pairs that preserve knowledge integrity. 2.2 Knowledge Point Graph Construction Knowledge Point Extraction and Consolidation. We define KPs as the fundamental units of content within a discipline, such as concepts, principles, theorems, or methods ( Duan et al. 2025 ; Hao et al. 2022 ) . For efficient and accurate annotation, we distill labeled data from DeepSeek-R1 and fine-tune the Qwen2.5-14B-Instruct model ( Qwen et al. 2025 ) to serve as our extractor (see Appendix C.1 ). We further consolidate the extracted KPs (see Appendix B.1 ), resulting in a final set of 10M high‑quality KPs. Notably, 75.43% of QA instances examine multiple KPs, indicating strong interrelations among KPs and motivating us to construct a KP graph with strong edge associations. Knowledge Point Graph Construction. The KP graph is built from the annotated QA dataset A=\{D_{i}\}_{i=1}^{m} , where each item D_{i}=(t_{i},s_{i},h_{i},K_{i}) consists of the