D-SCoRE: D ocument-Centric S egmentation and Co T Reasoning with Structured E xport for QA-CoT Data Generation
Weibo Zhou, Lingbo Li, Shangsong Liang · 2025
Read D-SCoRE: D ocument-Centric S egmentation and Co T Reasoning with Structured E xport for QA-CoT Data Generation by Weibo Zhou, Lingbo Li, Shangsong Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01309.txt.
Introduction
Weibo Zhou, Lingbo Li, and Shangsong Liang present a way to make training data for language models in a 2025 paper. Good question and answer sets for narrow fields are scarce and costly to build by hand. Their frame, called D-SCoRE, needs no training of its own and works from any text source.
The frame cuts a document into parts, writes questions with step by step reasoning, and exports the set in a clean format. Controls vary the roles asked about, balance question types, and add what if twists. Models trained on the made data were tested against other sets. The aim is cheaper, richer data tuned to a field.
Catalog record
| CREATOR | Weibo Zhou, Lingbo Li, Shangsong Liang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01309
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
D-SCoRE: D ocument-Centric S egmentation and Co T Reasoning with Structured E xport for QA-CoT Data Generation Weibo Zhou, Lingbo Li, Shangsong Liang Abstract The scarcity and high cost of high-quality question-answering (QA) datasets hinder supervised fine-tuning (SFT) for domain-specific large language models (LLMs). To address this, we introduce D-SCoRE, a training-free pipeline that utilizes LLMs and prompt engineering to produce diverse, high-quality QA datasets from arbitrary textual sources. D-SCoRE integrates D ocument-centric processing, S egmentation, Co T R easoning, and structured E xport to generate QA-CoT datasets tailored for domain-aware SFT. Multi-dimensional control mechanisms, such as semantic role transformation, question type balancing, and counterfactual materials, enhance diversity and relevance, overcoming limitations of existing QA generation. LLMs fine-tuned on D-SCoRE-generated QA datasets, and human-annotated QA datasets (SQuAD, Covid-QA) are evaluated on SQuADShifts and Covid-QA test sets, with D-SCoRE outperforming across most domains. D-SCoRE generates six QA-CoT pairs with four-option counterfactual materials per 100–200-word text in 90 seconds using an 8B LLM on consumer-grade hardware. Its simplicity and scalability enable efficient QA generation and high-performance fine-tuning across domains. Introduction Question-answering (QA) datasets are vital for supervised fine-tuning (SFT) of large language models (LLMs), enabling adaptation to domains such as education, healthcare, and customer service. However, high-quality QA datasets are scarce due to the labor-intensive and costly process of manual annotation, which limits scalability. While automated QA generation offers a promising solution, existing methods struggle with insufficient diversity, high production costs, and poor control over question complexityhighlighting the need for more efficient and versatile approaches. Existing automated QA generation methods can be classified into two categories: non-synthetic and semi-synthetic. Non-synthetic approaches extract QA pairs directly from corpora, ensuring contextual relevance but lacking scalability due to reliance on pre-annotated sources. Semi-synthetic methods augment human-curated data with automated processes, improving efficiency but often requiring complex preprocessing pipelines. Both paradigms face challenges in achieving diversity, cost-effectiveness, and complexity control, necessitating simpler and more controllable frameworks. To overcome these limitations, we propose a novel, training-free pipeline that generates diverse, high-quality QA datasets from arbitrary texts using advanced prompt engineering with LLMs. Our approach employs multi-dimensional control strategies—including explicit and implicit QA formulation, semantic role transformations, and counterfactual reasoning—to enhance dataset variety, accuracy, and relevance while reducing issues like hallucinations. Key contributions include: • A simplified, end-to-end framework that directly produces QA pairs from input texts, bypassing complex preprocessing or model fine-tuning. • The integration of semantic role transformations and counterfactual elements to diversify question types and increase complexity, thereby addressing gaps in existing semi-synthetic and fully synthetic methods. • Empirical validation showing superior performance of LLMs fine-tuned on our datasets compared to baselines, demonstrating practical efficacy in multi-domain applications. Related Work QA Dataset Generation The generation of QA datasets can be broadly categorized into non-synthetic and semi-synthetic paradigms. Non-synthetic approaches, such as SQuAD ( Rajpurkar et al. 2016 ) , rely on crowd-sourced questions derived from sources like Wikipedia articles, yielding high-quality annotations but incurring significant labor costs and scalability challenges. Similarly, domain-specific datasets like FairytaleQA ( Xu et al. 2022 ) provide detailed annotations tailored to areas such as children’s narratives, yet they remain resource-intensive to construct. Traditional semi-synthetic methods enhance efficiency by incorporating automated components, including answer-first ( Alberti et al. 2019 ) or question-first ( Rodriguez-Torrealba, Garcia-Lopez, and Garcia-Cabot 2022 ) strategies, often augmented with auxiliary signals ( Liu et al. 2020 ; Puranik, Majumder, and Chaoji 2023 ) . As categorized by Ushio, Alva-Manchego, and Camacho-Collados 2023 , these methods encompass pipeline-based, multitask, and end-to-end frameworks; however, they frequently exhibit limited flexibility due to reliance on intricate preprocessing steps. Contemporary semi-synthetic techniques increasingly harness LLMs for QA pair synthesis, employing strategies such as data augmentation ( Chowdhury and Chadha 2024 ) , optimized workflows ( Yuen et al. 2025 ) , and ablation analyses in structured domains like mathematics and programming, as explored in OpenThoughts ( Guha et al. 2025 ) . Additionally, template-based methods like Self-Instruct ( Wang et al. 2023 ) facilitate generation, but these approaches often fall short in achieving precise control over complexity, diversity, and factual fidelity, particularly in ungrounded or expansive contexts. QA Data Quality Enhancement Efforts to enhance QA dataset quality focus on improving accuracy, richness, and diversity. To bolster accuracy, techniques extract implicit relations ( Beckerman and Christakis 2019 ) , leverage summarization ( Dugan et al. 2022 ) , or incorporate structured planning mechanisms ( Li and Zhang 2024 ) , thereby enhancing relevance; nevertheless, these methods tend to be computationally demanding or exhibit limited generalizability. Richness is augmented through the inclusion of explicit or implicit question annotations, domain-adapted attributes ( Xu et al. 2022 ) , distractors, or multiple-choice formats ( Rodriguez-Torrealba, Garcia-Lopez, and Garcia-Cabot 2022 ; Zhou, Hu, and Wang 2025 ) , although such enhancements seldom address deeper semantic variability. Diversity, essential for developing robust QA models ( Sultan et al. 2020 ) , is pursued via implicit mechanisms (e.g., stochastic decoding ( Yadav et al. 2024 ) ) and explicit interventions (e.g., entity substitutions ( Paranjape, Lamm, and Tenney 2022 ) , conditioning on document positions, question categories, or named entities ( Yadav et al. 2024 ) , or iterative refinement processes ( Eo et al. 2023 ) ). Despite these advances, prevailing methods grapple with constraints in semantic control, redundancy mitigation, fluency preservation, rigid partitioning, or architectural complexity, which collectively impede scalability. Downstream Applications QA datasets underpin LLM fine-tuning, performance evaluation, and practical deployments. In training contexts, they enable supervised and parameter-efficient fine-tuning techniques (e.g., LoRA), as demonstrated in OpenThoughts’ refinement of Qwen2.5-7B-Instruct on reasoning benchmarks ( Guha et al. 2025 ) ; however, elevated curation expenses constrain broader adoption. For evaluation, benchmarks such as MMLU ( Hendrycks et al. 2020 ) and GSM8K ( Cobbe et al. 2021 ) assess multifaceted capabilities, while datasets like HotpotQA ( Yang et al. 2018 ) probe zero-shot and few-shot generalization, and VQA-RAD ( Lau et al. 2018 ) examines multimodal reasoning. Nonetheless, their domain-specific formulations often limit transferability. In applied settings, including education (e.g., FairytaleQA for narrative understanding ( Xu et al. 2022 ; Yao et al. 2022 ) ), healthcare (e.g., synthetic clinical QA for diagnostic support ( Bai et al. 2024 ) ), and conversational systems, challenges persist in the form of insufficient diversity, restricted scope, and inherent biases, thereby obstructing the development of resilient, domain-agnostic frameworks. These persistent limitations—encompassing suboptimal scalability, variable accuracy, constrained diversity, and diminished interpretability—underscore the necessity for an automated, versatile QA generation paradigm that delivers high-fidelity outputs, semantic depth, and broad applicability, independent of human annotations or elaborate infrastructures. Methodology This section delineates our methodology for synthesizing high-quality and diverse QA datasets tailored for SFT of LLMs. The proposed framework harnesses LLMs in conjunction with sophisticated prompt engineering to derive QA pairs directly from unprocessed textual inputs, obviating the necessity for supplementary training regimens or intricate architectural designs. Central to this approach is a tripartite core pipeline, fortified by multifaceted control mechanisms to augment diversity and fidelity. Supplementary modules may be integrated optionally to accommodate heterogeneous inputs and diverse downstream utilities. The methodology is engineered for simplicity, scalability, and cross-domain adaptability. Figure 1: Overview of D-SCoRE. Core Pipeline At the heart of our methodology