KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari, Mahdi Noori, Farhan Farsi, Behnam Bahrak Abstract With the rise of large language models (LLMs), they have become instrumental in applications such as Retrieval-Augmented Generation (RAG). Yet evaluating these systems remains bottlenecked by the time and cost of building specialized assessment datasets. We introduce KNIGHT, an LLM-based, knowledge-graph-driven framework for generating multiple-choice question (MCQ) datasets from external sources. KNIGHT constructs a topic-specific knowledge graph, a structured, parsimonious summary of entities and relations, that can be reused to generate instructor-controlled difficulty levels, including multi-hop questions, without repeatedly re-feeding the full source text. This KG acts as a compressed, reusable state, making question generation a cheap read over the graph. We instantiate KNIGHT on Wikipedia/Wikidata, while keeping the framework domain- and ontology-agnostic. As a case study, KNIGHT produces six MCQ datasets in History, Biology, and Mathematics. We evaluate quality on five criteria: fluency, unambiguity (single correct answer), topic relevance, option uniqueness, and answerability given the provided sources (as a proxy for hallucination). Results show that KNIGHT enables token- and cost-efficient generation from a reusable KG representation, achieves high quality across these criteria, and yields model rankings aligned with MMLU-style benchmarks, while supporting topic-specific and difficulty-controlled evaluation. 1 Introduction Recent work identifies two main levers for LLM progress: model size and data [ 27 ] . Because scaling parameters increases financial and environmental costs (e.g., CO 2 emissions) [ 32 , 17 ] , attention is shifting to dataset curation; yet expert-quality datasets are expensive and slow to build, and in applied settings such as RAG and task-specific fine-tuning, public evaluation datasets remain scarce due to proprietary data despite available toolkits [ 40 , 15 ] . Prior dataset-generation efforts exist [ 66 , 54 , 6 ] , but there is still no widely adopted open-source framework that is reproducible and easy to implement. Moreover, standard MCQ benchmarks such as MMLU [ 21 ] are largely static, difficult to update, provide limited instructor-level control over per-topic difficulty, and do not expose multi-hop structure for curriculum customization. In contrast, KNIGHT enables low-cost generation of topic-specific MCQ sets with user-controlled difficulty and explicit multi-hop design, while yielding model rankings aligned with MMLU-style [ 22 ] benchmarks. We introduce K nowledge-graph-driven N atural I tem G eneration with A daptive H ardness T uning ( KNIGHT ), a fully automated framework for synthesizing large-scale MCQ datasets from external document collections and ontologies with controllable difficulty. Given a user topic \tau (and optional prompt), KNIGHT runs four stages: (i) construct a topic-specific knowledge graph (KG) via retrieval-augmented extraction [ 31 , 38 , 19 ] , where the KG is a compact, parsimonious summary of entities and relations distilled from the sources; (ii) generate source-grounded MCQs by traversing multi-hop KG paths with configurable depth; (iii) calibrate difficulty based on path length and abstraction, validated via entropy-based uncertainty measures and human error patterns; and (iv) filter items with an LLM- and rule-based validator enforcing five criteria: grammar, single-correct-answer unambiguity, option uniqueness, answerability from evidence, and topicality [ 16 , 58 ] . KNIGHT integrates RAG-based extraction, KG-guided multi-hop generation, and LLM-based validation into a reusable, modular pipeline that caches a compact topic KG for efficient dataset creation, supports forward/reverse question modes, and uses human and entropy-based checks to mitigate imperfect answerability/difficulty proxies. Figure 1: KNIGHT High-level pipeline. Given a prompt/topic and depth, KNIGHT retrieves evidence, builds a focused KG, generates MCQs, and filters them to produce the final dataset. We treat answerability from retrieved evidence as a proxy for generator hallucination: items judged unanswerable from the sources indicate unsupported or hallucinated content. Since the KG is built once per topic and reused across many generations, KNIGHT enables token-efficient, low-cost generation from a reusable KG representation versus naive prompting that repeatedly re-ingests long evidence contexts per question. We use GPT-4o-mini for all LLM calls throughout, yielding a cost- and token-aware evaluation setting. We instantiate the type system \phi on the Wikipedia/Wikidata ontology (though in principle \phi can be defined over any domain ontology or enterprise schema) and introduce KNIGHT , a token-efficient, KG-driven pipeline for generating difficulty-controlled MCQ datasets from external sources and ontologies. As case studies of its flexibility and reusability (rather than standalone benchmark contributions), we construct six Wikipedia/Wikidata-based MCQ datasets spanning history, biology, and mathematics at two difficulty levels, and run four ablations alongside full KNIGHT using five GPT-4o-mini configurations (Plain, RAG, RAG+KG, RAG+Val, and full KNIGHT ). This staged comparison isolates how grounding, KG guidance, and validation affect hallucination, distractor quality, and difficulty calibration via automatic, human, and entropy-based evaluations. Items are generated within minutes on Google Colab T4 and are grammatical and difficulty-calibrated (Section 4 ). KNIGHT reduces hallucinations relative to Plain and RAG (Section A ) and yields model rankings aligned with MMLU-style benchmarks, while being cheaper and easier to update than static MMLU-like test sets, supporting it as a scalable, low-cost benchmark generator. Our code and package are publicly available on PyPI 1 1 1 https://pypi.org/project/knight-mcq/ and GitHub 2 2 2 https://github.com/ErfanShm/knight-mcq . 2 Related Work Knowledge Graph Construction. Constructing KGs from unstructured text typically uses multi-stage NLP pipelines (e.g. entity extraction/linking and relation extraction), often assuming a predefined schema and substantial supervision/training data [ 74 , 23 ] . Traditional systems commonly perform named entity recognition and model-based relation extraction to identify entities and relationships, but these often require predefined schemas and extensive training. Recent LLM-based methods reduce these requirements and improve portability: 31 propose iText2KG , a zero-shot incremental framework with LLM-powered entity/relation extraction for topic-independent KG construction; 13 extract triples from scientific abstracts via NLP/text-mining and integrate them into a KG; and 75 evaluate GPT-4 [ 49 ] on KG tasks, finding that it excels at reasoning, and introduce AutoKG , a multi-agent LLM approach with external retrieval. Prompting has also improved relation extraction, e.g., Wikidata-informed prompts in 36 . While we instantiate our type system using Wikipedia/Wikidata as an ontology [ 67 ] , the mapping function \phi can in principle be defined over other domain ontologies or schemas (e.g., enterprise KGs or specialized KBs); here we evaluate only the Wikipedia/Wikidata instantiation and leave broader generalization to future work. Finally, in line with retrieval-augmented generation, RAG combines parametric LMs with retrieved knowledge bases; 38 show it yields more specific, diverse, and factual outputs than parametric-only models. Question Generation from Knowledge Graphs and Structured Data. Early work generates natural questions from KG triples using keyword extraction and RNNs [ 55 ] . Later methods go beyond single triples by encoding subgraphs with Graph2Seq and copy mechanisms [ 9 ] , and by using contextual KGs with answer-aware GATs for coherent multi-hop question generation [ 39 ] . Difficulty control has also been studied explicitly: 29 condition multi-hop generation on estimated KG difficulty, while 10 guide reasoning complexity via step-by-step rewriting. Beyond graph-centric pipelines, LIQUID builds QA datasets directly from text through summarization, entity extraction, and question generation [ 37 ] . Evaluation and Filtering of Generated Questions. Ensuring the quality of generated questions requires multi-faceted evaluation, and recent work applies dedicated QA evaluation metrics. High-quality MCQs require multi-faceted evaluation: 44 survey metrics including LM perplexity, lexical diversity, grammar error rates, cognitive complexity, and answerability to assess fluency, uniqueness, and inferability, while 61 highlight semantic diversity gains from preference-tuned LLMs. Beyond these quality dimensions, factuality and safety remain concerns: even strong LLMs can hallucinate and may exhibit biases, motivating automatic filtering and validation when generating educational content [ 57 ] . Factuality is further challenged by the tendency of LLMs to answer confidently even when inputs are unanswerable, motivating explicit answerability checks as a practical proxy for hallucination control [ 62 ] . In RAG, recent work also evaluates whether systems correctly reject unanswerable requests, complementing accuracy on answerable ones [ 52 ] . Finally, LLM-based review/validator pipelines can automatically assess MCQ validity across multiple criteria, reducing reliance on purely manual screening [ 45 ] . Building on these lines, we propose KNIGHT , a unified end-to-end framework that integrates KG construction, graph-driven question generation, and automatic quality filtering. A user-defined difficulty parameter controls graph depth to elicit multi-hop or higher-order items, while LLMs both generate and validate MCQs from KG paths. Compared to prior pipelines, our approach emphasizes reusable, token-efficient KG representations and a comprehensive LLM-powered evaluation/validation stack, aiming to produce diverse, high-quality QA pairs with improved reliability for topic-specific question sets. 3 System Design 3.1 Knowledge-Graph Constructor Figure 2: KNIGHT architecture. (Left) A topic/prompt-driven RAG pipeline retrieves evidence, extracts triples, and curates a compact KG under depth budget d_{\max} ; (Right) multi-hop paths are sampled to generate questions/distractors and validated for evidence-grounded answerability to form the final MCQA dataset. Given a user-specified topic \tau , optional prompt, and hardness budget d_{\max}\in\mathbb{N} , the Knowledge-Graph Constructor builds a directed property graph G=(V,E,\mathcal{R}) with canonicalized entities v\in V and labeled edges (v_{h},r,v_{t})\in E , r\in\mathcal{R} . It iterates a retrieve–generate–filter loop (Alg. 1 ) combining external retrieval with LLM reasoning; the backend is swappable (HuggingFace-compatible) via a config flag. Parsimonious, reusable representation. Once built, the KG can be cached and reused to generate many difficulty-controlled question sets (varying hop length, formats, and targets) without re-feeding long source documents, amortizing the one-time construction cost and improving token efficiency. Evidence retrieval and description synthesis. We first retrieve a ranked context \mathcal{D}=\{d_{1},\dots,d_{k}\} from Wikipedia (or other open sources) using dense passage retrieval and re-ranking [ 38 , 19 ] . Conditioned on \tau (and Algorithm 1 GraphGenerator —depth-bounded KG construction 1: seed topic/entity v_{0} , depth limit d_{\max} 2: G\leftarrow empty graph; G.\text{addNode}(v_{0}) 3: Q\leftarrow[(v_{0},0)] \triangleright FIFO over (node,depth) 4: while Q\neq\emptyset do 5: (v,\ell)\leftarrow\textsc{PopFront}(Q) 6: \mathcal{D}\leftarrow\textsc{Retrieve}(v) 7: \delta\leftarrow\mathcal{L}_{\mathrm{desc}}(v,\mathcal{D}) 8: R\leftarrow\mathcal{L}_{\mathrm{rel}}(\delta) 9: C\leftarrow\textsc{Curate}(R) 10: for all (v,r,u)\in C do 11: G.\text{addNode}(u) ; G.\text{addEdge}(v,r,u) 12: if \ell+1\leq d_{\max} then 13: \textsc{PushBack}(Q,(u,\ell+1)) 14: end if 15: end for 16: end while 17: return G the optional prompt) and \mathcal{D} , the Description Generator \mathcal{L}_{\mathrm{desc}} produces a structured eight-point gloss \delta (Appendix D ). Triple induction and deduplication. The Relation Extractor maps \delta to a triple set R(\delta) (Eq. 3.1 ) and removes near-duplicates using a Levenshtein filter with threshold \lambda_{\max} [ 31 , 13 ] , then passes the remaining candidates to the Curator. Curation, pruning, and depth control. The Curator applies (i) type checks (instantiated here with Wikidata), (ii) NLI-based consistency checks between node glosses and relation statements [ 48 , 7 , 69 ] , and (iii) content-policy screening following prior safety analyses [ 58 ] . Graph expansion proceeds breadth-first and stops at depth d_{\max} , yielding the bounded neighborhood V_{d_{\max}}=\{\,v\mid\mathrm{dist}_{G}(v_{0},v)\leq d_{\max}\,\} (Eq. 3.1 ), which matches the KG scope to the downstream MCQ generator (Section 3.2 ). KG-1: Retrieval-Augmented Description Synthesis. Figure 2 shows the first stage: producing an eight-point gloss \delta(v_{0}) for the seed v_{0} via a rank-and-generate RAG pipeline [ 38 , 19 ] with (i) a dense retriever \mathcal{R}_{\mathrm{enc}} (Contriever base; 25 ) encoding topic \tau and scoring against a BM25-filtered corpus [ 59 ] , and (ii) a cross-encoder re-ranker \mathcal{R}_{\mathrm{rer}} (MiniLM-L12; 56 ) refining the top–50 to k{=}5 passages \mathcal{D}_{0}{=}\{d_{1},\dots,d_{5}\} with scores s(d_{i})\!\in\![0,1] . Each retained passage d_{i} is injected into system-prompted GPT-4 o -mini , yielding a candidate description d_{i}^{\star}=\mathcal{L}_{\mathrm{desc}}(\tau,d_{i}) ; to combine evidence we model the generation probability as a RAG mixture [ 38 ] : P(d\mid\tau)=\sum_{z\in\mathcal{D}_{0}}P_{\theta}(d\mid\tau,z)\underbrace{\frac{\exp s(z)}{\sum_{z^{\prime}}\exp s(z^{\prime})}}_{P_{\text{ret}}(z\mid\tau)}, (1) where P_{\theta} is parameterised by GPT-4 o -mini; if \mathcal{D}_{0}=\varnothing (no scores >\!0.15 ), we fall back to parametric generation P(d\mid\tau){=}P_{\theta}(d\mid\tau) . We retain a node gloss only if it is traceable to at least one retrieved passage, using: \gamma(\delta)=\begin{cases}1&\exists z\in\mathcal{D}_{0}:\texttt{overlap}(z,\delta)\geq\eta,\\[4.0pt] 0&\text{otherwise},\quad\eta=0.35,\end{cases} (2) discarding \gamma(\delta)=0 descriptions; this makes persistent node content externally verifiable, mitigating hallucination risk [ 26 ] . The validated description \delta(v_{0}) is then forwarded to the Relation Extractor (§ 3.1 ), enabling breadth-first expansion up to depth d_{\max} (Algorithm 1 ). KG-2: Triple Induction via Relation Extraction. Given a stored gloss \delta in description , we distil explicit facts using the extractor \mathcal{L}_{\text{rel}} , implemented with GPT-4 o -mini 3 3 3 Checkpoint gpt_4o_mini_2024_05 . , which is prompted to emit a JSON list of (\textit{head},\textit{relation},\textit{tail}) triples. Formally, R(\delta)=\{(h,r,t)\mid h,t\in\mathcal{E},\,r\in\mathcal{R}\} , where \mathcal{E} is the dynamic entity inventory and \mathcal{R} a controlled relation schema (cf. 31 , 13 ). A Levenshtein filter with threshold \lambda_{\max} removes near-duplicate triples before insertion. KG-3: Depth-Controlled Expansion (token-efficient). Graph growth is bounded by the hardness budget d_{\max} : we run breadth-first expansion with a FIFO queue over (v,\ell) (Algorithm 1 ), re-applying KG-1–KG-2 while \ell Memory can be used instead of Neo4j instance by passing backend="memory" to the constructor which results in a non-persistent KG storage. For item synthesis we default to GPT-4o-mini. The API key of LLM must be provided: $ export OPENAI_API_KEY= sk-•••• Any HuggingFace-compatible decoder (e.g., Llama-2 [ 65 ] ) can be hot-swapped by setting lm_backend="hf" . C.2 Unified Workflow (API & CLI) Both interfaces expose identical functionality; we illustrate each workflow with a minimal depth- \!2 example that produces ten MCQs on Biology . ⬇ from knight import KnightFramework as kframe kf = kframe ( uri = "bolt://localhost:7687" , user = "neo4j" , password = "neo4j" ) kf . build_kg ( topic = "Biology" , depth =2) ds = kf . generate ( prompt = "multiple-choice" , topic = "Biology" , depth =2, num_q =10) report = kf . validate ( ds ) ds . to_json ( "bio_d2.json" ) Now for the CLI we have: $ knight --topic "Biology" --prompt "multiple-choice" --depth 2 --num-q 10 --output bio_d2.json --validate C.3 Advanced Settings All components are plug-and-play: alternative KGs (e.g., Wikidata), relation whitelists, or custom prompt templates can be swapped without touching core logic, promoting reproducible ablations. Full configuration options for replication are available in our README.md . In sum, KNIGHT combines structured knowledge retrieval with controllable LLM generation to deliver fact-grounded, difficulty-calibrated MCQ datasets, suitable for both educational deployment and rigorous LLM evaluation. Appendix D Prompts This section documents the prompts employed in our study, with particular emphasis on few-shot prompting techniques that enable models to perform novel tasks without parameter updates [ 8 ] . Each prompt consists of a title showing to which process it belongs and whether it is a system prompt or a user prompt, a purpose explaining its usage, and content. Structured Term Explanation System Prompt Purpose: Sets the LLM’s persona as a scientific subject-matter expert and defines a required 8-point structure for generating comprehensive term explanations. Content: You are a subject-matter expert in a scientific field. Your task is to provide detailed, thorough, and academically structured explanations about terms provided by the user. Each term should be explained exhaustively using the following structure: 1. Definition and Scope – Provide a precise, scientific definition of the term. Outline its general scope, including the boundaries and extent of its meaning and use. 2. Domains of Use – Identify all relevant scientific, technical, or professional domains where this term plays a key role. Specify the fields in which this concept is critical and explain its importance in each. 3. Subfields and Disciplines – Break the term down into its major subfields, branches, or areas of study. Provide a brief but comprehensive overview of each subfield, including key principles, practices, and contributors. 4. Key Concepts and Mechanisms – Describe the most important ideas, mechanisms, or processes associated with this term in various contexts. Explain how these ideas interconnect. 5. Real-World Applications – Discuss the major practical applications of this concept in different spheres, such as industry, healthcare, environmental science, etc. 6. Case Studies and Examples – Provide specific case studies, examples, or practical demonstrations of the term in action. Show how it is applied in real-world scenarios. 7. Related and Overlapping Terms – Identify related or similar terms and concepts. Clarify how they are connected, and explain any subtle distinctions. 8. Current Research and Trends – Briefly cover the current research directions, innovations, and debates around this concept. Mention any ongoing advancements or challenges in the field. Your explanation should be clear, well-organized, scientifically accurate, and educational. Assume that the user is unfamiliar with the term, so explain each concept thoroughly. Use precise language and cite notable research, when possible. Dive deeply into subtopics as needed to provide a full understanding of the term’s scope and implications. Structured Term Explanation User Prompt Purpose: Used when an unambiguous Wikipedia summary is found. It instructs the LLM (paired with the System Prompt above) to generate the structured explanation for a specific “ [term] ”, using the “ [wikipedia_summary] ” as the primary source and optionally considering the “ [parent_term] ”. Content: Now, please apply the structured explanation approach defined in the system prompt to explain the term: “ [term] ”. Use the following Wikipedia context as the primary source for your explanation, structuring your response according to the system prompt guidelines: --- Wikipedia Context --- ‘‘[wikipedia_summary]’’ --- End Wikipedia Context --- Also consider its relationship to the parent term “ [parent_term] ”. (Note: The last line regarding “ [parent_term] ” is conditional) Wikipedia Title Relevance Check System Prompt Purpose: Sets the context for the LLM, telling it to act as a relevance classifier or "domain-specific semantic filter". It needs to decide if a given Wikipedia page title is a good source for defining a specific term, considering the provided context. It explicitly asks for a "Yes" or "No" answer only. Content: You are performing a relevance classification task to evaluate whether a Wikipedia page title is an appropriate definition source for a given term within a specific context. You are expected to act as a domain-specific semantic filter. Answer "Yes" only if the title refers directly to the term and aligns with the context. If the title is ambiguous, only tangentially related, or contextually irrelevant, answer "No". Respond with only one word: "Yes" or "No". Wikipedia Title Relevance Check User Prompt Purpose: Provides the specific data for the LLM to evaluate: the “ [term] ” needing definition, the “ [title_guess] ” (candidate Wikipedia page title), and the “ [context_hint] ” (which could be a parent term, source text, or "general knowledge"). It reiterates the request for a ’Yes’ or ’No’ answer. Content: Context: Information related to “ [context_hint] ”. Term to define: “ [term] ”. Candidate Wikipedia Page Title: “ [title_guess] ”. Evaluate relevance and respond with only ’Yes’ or ’No’. Forward MCQ Generation System Prompt Purpose: Instructs the LLM to act as a "structured question generation system". Its goal is to create an MCQ (question, 4 options, correct answer key) based on a multi-step path provided from a knowledge graph. The question should require reasoning across the path, and the answer should be implied by the path details. Content: You are a structured question generation system. Your task is to generate a question and a concise answer based on a multi-hop path in a knowledge graph and node descriptions. The question must reflect reasoning over the multi-step relationships in the path. The answer should be clearly implied by the path and descriptions, often referring to a specific node. Forward MCQ Generation User Prompt Purpose: Provides the LLM with the specific details needed to generate the forward MCQ: examples of the task, the actual graph “ [path_representation] ”, descriptions of the “ [start_node] ” and “ [end_node] ”, an optional “ [topic] ” constraint, and strict formatting instructions for the output (Question, A, B, C, D, Correct Answer key). Content: Follow the instructions in the system prompt to generate a multiple-choice question based on the provided path and node descriptions. --- Few-Shot --- ‘‘[few-shot example]’’ --- End Few-Shot --- IMPORTANT: The generated Question and Options MUST be relevant to the overall topic: “ [topic] ”. Now, generate for the following: Path: ‘‘[path_representation]’’ Start Node: ‘‘[start_node]’’ Description: ‘‘[start_desc]’’ End Node: ‘‘[end_node]’’ Description: ‘‘[end_desc]’’ IMPORTANT: You MUST generate exactly four options (A, B, C, D) and indicate the single correct answer key. Adhere strictly to the output format below. Output: Question: [Your generated question reflecting the multi-step path] A) [Option A] B) [Option B] C) [Option C] D) [Option D] Correct Answer: [A, B, C, or D] Reverse MCQ Generation System Prompt Purpose: Sets the LLM’s role as a "reasoning assistant" focused on generating reverse questions. The goal is to create an MCQ where the “ [start_node] ” of the provided graph path is the correct answer. It suggests using the end node’s perspective to guide the reasoning. Content: You are a reasoning assistant generating reverse questions from knowledge graph paths. Your task is to generate a question that can be answered explicitly by the start node of a multi-hop path. Use the end node’s perspective when possible to guide the reasoning backward. Reverse MCQ Generation User Prompt Purpose: Provides the LLM with specific instructions and data to generate the reverse MCQ. It includes examples, the graph, descriptions of “ [start_node] ” and “ [end_node] ”, an optional “ [topic] ” constraint, and strict formatting instructions. Crucially, it emphasizes that the correct answer must be the “ [start_node] ”. Content: Follow the instructions in the system prompt to generate a multiple-choice question where the start node (“ [start_node] ”) is the correct answer. --- Few-Shot --- ‘‘[few-shot example]’’ --- End Few-Shot --- IMPORTANT: The generated Question and Options MUST be relevant to the overall topic: “ [topic] ”. Now, generate for the following: Path: ‘‘[path_representation]’’ Start Node: ‘‘[start_node]’’ Description: ‘‘[start_desc]’’ End Node: ‘‘[end_node]’’ Description: ‘‘[end_desc]’’ IMPORTANT: You MUST generate exactly four options (A, B, C, D) and indicate the single correct answer key (which MUST correspond to the option containing the Start Node name “ [start_node] ”). Adhere strictly to the output format below. Output: Question: [Generated question targeting the start node] A) [Option A] B) [Option B] C) [Option C] D) [Option D] Correct Answer: [Letter corresponding to the option containing the exact text “ [start_node] ”] GPT Triplet Extraction System Prompt Purpose: This prompt instructs the LLM to extract significant subject-predicate-object triplets from the provided text. It gives detailed guidelines on what to focus on (key concepts, important relationships) and what to ignore (pronouns, generic terms). It specifies the required JSON output format and provides clear examples of good and bad triplets. Content: You are an information-extraction specialist. Extract only the most significant and meaningful “ [subject–predicate–object] ” triplets from any text you receive. Here are the guidelines you should follow : • Focus on important entities: names, places, concepts, achievements. • Include defining characteristics and significant relationships. • Capture major influences, contributions, and key life events. • Skip generic pronouns, articles, and common words. • Write relations in clear lowercase and with underscores. IMPORTANT: The generated output must accommodate with this format. { "triplets": [ { "head": "specific_entity", "relation": "significant_relation", "tail": "important_concept" }, { "head": "major_figure", "relation": "notable_achievement", "tail": "specific_contribution" } ] } Bellow are some of the good and bad examples: --- Few-Shot --- ‘‘[few-shot example]’’ --- End Few-Shot --- GPT Triplet Extraction User Prompt Purpose: Provides the LLM with the specific text to extract triplets based on the instructions given in system prompt. Content: Follow the instructions in the system prompt to extract subject-predicate-object triplets from the text below. --- Start of the text input --- ‘‘[text-content]’’ --- End of the text input --- MCQ Validation System Prompt Purpose: Defines the LLM’s role as an evaluator for MCQs generated from knowledge graph paths. It needs to assess grammar/clarity, whether the correct answer key is supported only by the provided path details, and optionally, relevance to a given topic. It demands a specific output format. Content: You are MCQ-validation assistant. Evaluate a four-option multiple-choice question (MCQ) using only the information supplied in the “Source Information” block. Answer with five “ [YES/NO] ” (or N/A) tags in the exact order and casing shown below. Checklist 1. GRAMMAR_FLUENCY Is the Question spelled and phrased correctly and clearly? 2. SINGLE_CORRECT_KEY Is exactly one option marked as correct? 3. OPTION_UNIQUENESS Are all four options distinct (no duplicates or near-duplicates)? 4. ANSWERABLE_FROM_SOURCE Does the indicated correct option follow solely from the Source (path, node excerpts) without outside knowledge? 5. TOPIC_RELEVANCE If a Topic is provided, is the MCQ clearly about that topic? MCQ Validation User Prompt Purpose: Provides the LLM with the specific MCQ data (“ [question] ”, “ [correct_answer_key] ”) and its source-details (including the “ [path_representation] ”, “ [start_node] ”, “ [end_node] ” and etc) to evaluate. It lists the evaluation criteria and specifies the required output format lines. Content: Follow the instructions in the system prompt to evaluate the following MCQ based only on the Source Information. --- Few-Shot --- ‘‘[few-shot example]’’ --- End Few-Shot --- Now, evaluate the following question: Question: “ [question_text] ” A) [Option A] B) [Option B] C) [Option C] D) [Option D] Correct Answer: “ [correct_answer_key] ” Topic (optional): “ [topic_or_blank] ” Source Information Path: ‘‘[path_representation]’’ Start Node: ‘‘[start_node]’’ Description: ‘‘[start_desc]’’ End Node ‘‘[end_node]’’ Description: ‘‘[end_desc]’’ IMPORTANT: You MUST generate exactly 5 responses for each criterion based on the provided output below. Output: Grammar_Fluency: “ [YES/NO] ” Single_Correct_Key: “ [YES/NO] ” Option_Uniqueness: “ [YES/NO] ” Answerable_From_Source: “ [YES/NO] ” Topic_Relevant: “ [YES/NO or N/A] ” Term Extraction System Prompt (Baseline pipeline) Purpose: Defines the LLM’s role as an expert at identifying key encyclopedic terms from a text and specifies strict JSON output requirements. Content: You are an expert at identifying key encyclopedic terms from a text. Extract only the most significant and specific terms from the provided text. These terms should be ideal candidates for a Wikipedia or encyclopedia lookup. Return your answer strictly in the JSON schema shown below. GUIDELINES 1. Focus on concrete nouns, named entities, and specific scientific concepts. 2. Keep the terms concise and specific. 3. Extract the base form of a term (e.g., cell” instead of cells”). 4. Ensure the entire output consists strictly of the JSON object, with no preceding or succeeding text. OUTPUT FORMAT (MANDATORY) "terms": [ "term1", "term2", "term3" ] EXAMPLES (GOOD) ✓ mitochondria” ✓ Gregor Mendel” ✓ photosynthesis” ✓ natural selection” AVOID (BAD) × various aspects” × complex functions” × scientific study of life” × living organisms” Appendix E Additional Evaluation Metrics The main paper focuses on grammatical fluency, presence of a single correct answer option, answerability, and topic-relevance. Here we document additional metrics that were computed for every dataset but omitted from the core discussion for space and interpretability reasons. E.1 Question Length Diversity Recent studies indicate that question length can significantly influence LLM accuracy. 5 found that longer medical exam questions were associated with lower model accuracy. Similarly, 1 reported that ChatGPT was more likely to answer longer USMLE-style questions incorrectly. 71 likewise observed that LLMs achieve significantly higher accuracy in shorter math word problems. Motivated by these findings, we ensured that our generated questions span a broad range of lengths. This design allows us to evaluate model performance on both short and long questions. Figure 3 shows the resulting distribution of question lengths for each dataset; notably, these distributions closely approximate a normal shape, indicating a balanced mix of short and long questions. Figure 3: Distribution of question lengths for each dataset (histograms), demonstrating an approximately normal shape. E.2 Formalizing High-Quality Four-Option MCQs A key contribution of this work is the precise specification and validation of five core criteria that distinguish a high-quality multiple-choice question (MCQ) with exactly four options. Below we restate each criterion, provide its formal definition, and illustrate compliant versus non-compliant examples. 1. Grammatical Fluency Ensures the stem and options are free from spelling or grammatical errors and read naturally. Formally, a question q satisfies this criterion if it passes both automated grammar checks and human inspection for clarity and style. We quantitatively assess grammatical accuracy of each question q comprising W words by detecting the number of grammatical errors E using the LanguageTool 4 4 4 GitHub for LanguageTool system [ 35 , 34 ] . The Grammar Quality Score is defined as \text{GrammarQuality}(q)=1-\frac{E}{W} (4) which penalizes questions proportionally to the error frequency relative to length. For fluency evaluation, we employ the LangCheck toolkit [ 11 ] , which estimates naturalness via normalized log-probabilities from a pretrained language model. Higher fluency scores correspond to more coherent and natural text, a correlation empirically supported by 16 . Example of Non-compliance: Which of the following is the capital city of France? This question exhibits a grammatical error due to the omission of the definite article “the” and contains awkward phrasing. Compliant form: Which of the following is the capital city of France? This formulation demonstrates correct grammar and natural syntactic flow. 2. Single Correct Key Exactly one option o_{k}\in\{o_{1},o_{2},o_{3},o_{4}\} is correct: \exists!\,o_{k}:\;\mathrm{Correct}(o_{k})=\mathrm{True}. (5) This avoids ambiguity in scoring and interpretation. Example of Non-compliance: Which are prime numbers? Options: \{2,3,4,5\} (with two correct answers: 2 and 3). This violates the single-correct-key criterion due to multiple correct options. Compliant form: Which number is the smallest prime? Options: \{2,4,6,8\} (only one correct answer: 2) . This question satisfies the single-correct-key requirement by providing exactly one unambiguous correct option. 3. Option Uniqueness All distractors must differ sufficiently from each other. For options o_{i},o_{j} : \mathrm{sim}(o_{i},o_{j})<\delta, (6) where \mathrm{sim} is a lexical/semantic similarity metric and \delta a low threshold. Example of Non-compliance: Options: {“New York City”, “NYC”, “Los Angeles”, …}. The options include near-duplicate distractors (“New York City” and “NYC”), violating the option uniqueness criterion due to high lexical and semantic similarity. Compliant form: Options: {“New York City”, “Los Angeles”, “Chicago”, “Houston”}. The distractors are lexically and semantically distinct, satisfying the option uniqueness requirement by providing clearly differentiated answer choices. 4. Answerability from Source The correct answer must be derivable solely from the provided external knowledge G and question q : P(o_{k}\mid G,q)\;\gg\;P(o_{i}\mid G,q),\quad\forall i\neq k. (7) Example of Non-compliance: If G lacks “Eiffel Tower” data, asking “Where is the Eiffel Tower?” is invalid. Compliant form: If G contains “Paris is the capital of France,” asking “What is the capital of France?” is valid. 5. Topic Relevance Ensures semantic alignment with the specified domain topic T . We compute an entailment score: \begin{split}S(q,T)&=P(\text{entailment}\mid\\ &\quad\text{premise}=T,\text{ hypothesis}=q)\end{split} (8) Here a high entailment score indicates that the generated questions are strongly aligned with and highly pertinent to the specified topic. Example of Non-compliance: A photosynthesis question in “World History.” Compliant form: “What sparked the outbreak of World War I?” in “World History.” E.3 Quantitative Analysis of Expert Annotations and Quality Flags Dataset GRAMMAR SINGLE_KEY OPTION_UNIQUENESS ANSWERABLE TOPIC Hist-1 1 2 3 6 9 Bio-1 0 1 2 4 7 Math-1 2 2 2 5 8 Hist-3 2 2 3 6 10 Bio-3 1 2 1 4 4 Math-3 2 3 2 6 4 Table 6: Aggregated expert-raised flags indicating potential quality violations by dataset and criterion. Fewer than 5% of items trigger any flag, underscoring overall question quality. Our human evaluation protocol was carefully designed to maximize both reliability and validity, following established best practices in NLP evaluation studies. We recruited a total of thirty domain experts, organized into three groups of ten, each group specialized in one of the three dataset domains, to answer the benchmark questions. All thirty respondents were Iranian (nine female, twenty-one male), ranging in age from 29 to 54 years. None of these experts received any form of compensation; their participation was entirely voluntary, consistent with standard definitions of volunteer engagement. Each expert answered 40 questions from the dataset assigned to them. Because we had 10 experts per topic and two datasets per topic, this yields 5 experts per dataset; at 40 questions each, a total of 200 questions were completed for every dataset. This design balances workload while preserving annotation consistency. Annotators were given unlimited time and unrestricted access to relevant resources to ensure comprehensive, accurate responses. In addition to their primary assignments, 100 further questions per dataset were randomly sampled for quality auditing. All experts, beyond their 40 primary questions, reviewed and flagged 20 randomly sampled questions from each dataset according to our five core evaluation criteria. The consistency of flags and judgments across datasets indicates robust sampling and a well-distributed evaluation workload. Beyond measuring response accuracy, experts were instructed to flag any question exhibiting ambiguity or quality concerns across our five core evaluation criteria on the 100 random samples of each dataset, facilitating nuanced qualitative feedback alongside robust inter-annotator agreement analyses. Fluency annotations were conducted by a dedicated team of five additional experts, all Iranian (four male, one female), aged 25 to 41, with CEFR C1/C2 proficiency certifications. Importantly, all thirty-five participants (the thirty question-answerers plus the five fluency annotators) were fully briefed on the study’s objectives, provided informed consent, and were aware that their responses and annotations would be published. We computed the Pearson correlation coefficient r between human error rates E_{\mathrm{human}} and model entropy scores H across datasets and difficulty levels, obtaining: r=\frac{\operatorname{cov}(E_{\mathrm{human}},H)}{\sigma_{E}\sigma_{H}}\approx 0.78, (9) indicating a strong positive correlation between human-perceived difficulty and model uncertainty. Third, inter-annotator agreement, measured by Fleiss’ Kappa \kappa , consistently exceeded 0.82 in all domains: \kappa=\frac{\bar{P}-\bar{P}_{e}}{1-\bar{P}_{e}}>0.82, (10) where \bar{P} and \bar{P}_{e} denote observed and chance agreement, respectively. This confirms high annotation reliability. These findings affirm that our difficulty stratification, grounded in knowledge graph depth, meaningfully aligns with human cognitive assessments of question complexity. Moreover, the greater increase in model entropy from level 1 to 3 relative to the rise in human error rates suggests that large language models possess heightened sensitivity to subtle complexity variations, pointing toward promising directions for interpretability research. In addition to these quantitative measures, detailed quality control was conducted through expert-flagged quality violations across five criteria: grammar, single correct key, option uniqueness, answerability from source, and topic relevance. Experts evaluated questions thoroughly without strict time constraints, enabling rich qualitative feedback. Table 6 presents the aggregated counts of expert-raised flags per criterion and dataset. The relatively low incidence of flagged issues attests to the high linguistic correctness, and semantic validity of the generated datasets, even as difficulty increases. This strong human validation corroborates the effectiveness of our combined human-algorithmic quality assurance approach. Overall, the rigorous quantitative and qualitative quality validation presented here is critical for establishing trust in our datasets for downstream NLP tasks and benchmarking. It sets a replicable standard for future large-scale QA dataset construction, ensuring semantic rigor and interpretability. E.4 Significance Analyses for Topic Relevance Goal and Setting. We compare per-item topicality between our system ( KNIGHT ) and a GPT4o-mini baseline across subjects (History, Biology, Math) and difficulty levels (L1/L3). We evaluate two continuous topicality signals in [0,1] : (i) an MNLI-based entailment score that treats the topic as premise and the question as hypothesis; and (ii) an LLM-based topicality score computed via few-shot prompting. Higher is better in both. The baseline set includes about 100 items per split, and KNIGHT about 1{,}000 . Pre-processing and quality control. For each split and signal we validate bounds ( [0,1] ), remove exact item duplicates, and retain all remaining observations. We analyze each split separately; an “overall” roll-up is provided only for descriptive context and not as a substitute for per-split inference. What we test and why. We ask whether topicality differs meaningfully between systems. To cover complementary notions of difference we use: • Welch’s t for mean differences under unequal variances and unbalanced sample sizes. • Mann–Whitney U (MWU) and Brunner–Munzel (BM) for distributional differences robust to non-normality, ties, and unequal variances/shapes, critical under ceiling compression and strong n , imbalance. • A light 1% winsorized Welch as a sensitivity check to stabilize variance when many scores cluster near 1.0. • Effect sizes , Hedges’ g and Cliff’s \delta , to quantify practical differences; by convention, |g|\lesssim 0.2 and |\delta|<0.147 indicate small effects. Multiple comparisons. Within each test family (e.g., all Welch tests across the six splits per signal), we control family-wise error using Holm’s step-down procedure. Unless otherwise noted, “non-significant” refers to Holm-adjusted p values. Ceiling effects and n -imbalance: interpretive caveat. The baseline distributions are heavily compressed near 1.0 with very small variance, and the baseline sample size is much smaller ( \sim 100 vs. \sim 1,000 ) ; parametric standard errors can become unrealistically small even when mean gaps are tiny, sometimes making raw |t| look larger than warranted. Rank-based tests (MWU, BM) and effect sizes are therefore more reliable arbiters; we prioritize them alongside Holm-adjusted decisions. Results: MNLI-based entailment. Table 7 reports Welch, MWU, BM, effect sizes, and Holm-adjusted p per split and for a pooled “overall” summary. Across all splits, all Holm-adjusted p>0.05 and all effect sizes are small . Medians are nearly identical and both systems concentrate near the top of the scale. The winsorized Welch check does not change conclusions. Split (Entailment) Welch t ( p ) MWU ( p ) BM ( p ) Hedges’ g Cliff’s \delta Holm p History (L1) -0.98 ( 0.33 ) p=0.41 p=0.37 -0.10\;|\;-0.06 0.44 Biology (L1) -1.12 ( 0.26 ) p=0.49 p=0.44 -0.09\;|\;-0.05 0.52 Math (L1) -1.35 ( 0.18 ) p=0.38 p=0.31 -0.12\;|\;-0.07 \mathbf{0.18} History (L3) -1.28 ( 0.20 ) p=0.46 p=0.40 -0.11\;|\;-0.06 0.39 Biology (L3) -0.89 ( 0.37 ) p=0.52 p=0.48 -0.08\;|\;-0.04 0.60 Math (L3) -1.41 ( 0.16 ) p=0.35 p=0.29 -0.13\;|\;-0.07 0.21 Overall -1.47 ( 0.14 ) p=0.32 p=0.28 -0.12\;|\;-0.07 0.30 Table 7: Per-item topicality comparison (MNLI-based entailment). All tests are non-significant after Holm; effect sizes are uniformly small. The minimum Holm-adjusted p across splits is \approx 0.18 . Results: LLM-based topicality. Table 8 shows the same pattern: all Holm-adjusted p>0.05 and small Hedges’ g and Cliff’s \delta across splits. Nonparametric evidence again indicates no stochastic dominance. Sensitivity checks are consistent. Split (LLM) Welch t ( p ) MWU ( p ) BM ( p ) Hedges’ g Cliff’s \delta Holm p History (L1) -1.05 ( 0.29 ) p=0.43 p=0.39 -0.11\;|\;-0.06 0.46 Biology (L1) -0.92 ( 0.36 ) p=0.47 p=0.41 -0.10\;|\;-0.05 0.50 Math (L1) -1.22 ( 0.22 ) p=0.39 p=0.33 -0.12\;|\;-0.07 0.22 History (L3) -1.18 ( 0.24 ) p=0.45 p=0.40 -0.11\;|\;-0.06 0.41 Biology (L3) -0.71 ( 0.48 ) p=0.54 p=0.50 -0.08\;|\;-0.04 0.65 Math (L3) -1.36 ( 0.17 ) p=0.36 p=0.30 -0.13\;|\;-0.07 0.20 Overall -1.31 ( 0.19 ) p=0.34 p=0.30 -0.12\;|\;-0.07 0.29 Table 8: Per-item topicality comparison (LLM-based topicality score). All tests are non-significant after Holm; effect sizes are small. Integrated interpretation and consistency with main text. Across signals and splits, any apparent baseline edge in raw means is not supported once ceiling and n -imbalance are accounted for: (a) all Holm-adjusted p>0.05 (the smallest adjusted p observed is \approx 0.18 ), (b) effect sizes are uniformly small, and (c) rank-based tests do not indicate distributional shifts. This aligns with the main-text statement that “after Holm, all p>0.05 (min \approx 0.18 ),” and explains any residual large raw t as an artifact of ceiling/ n -imbalance rather than a practically meaningful gap. We therefore treat topicality as matched and focus the discussion on difficulty control, diversity, and validity. Reproducibility. For every split and signal we report sample sizes, means, standard deviations, medians, IQRs, Welch (statistic, d.f., p ), MWU ( U , tie-corrected p ), BM (statistic, p ), Hedges’ g , Cliff’s \delta , and Holm-adjusted p . The analysis code fixes seeds, applies identical pre-processing to both systems, and exports a complete per-item CSV plus a YAML manifest of test settings (winsorization, tie handling) to enable byte-identical re-analysis. Note (ceiling/ n -imbalance). Due to the combination of sample-size imbalance (e.g., \sim 100 baseline vs. \sim 1,000 in KNIGHT ) and near-zero baseline variance (ceiling effects), Welch’s t can inflate despite negligible practical gaps; we therefore ground conclusions in effect sizes, rank-based tests, and Holm-adjusted decisions. E.5 Visualizing Entropy Distributions We utilize a boxen plot combined with a swarm plot overlay (Figure 4 ) to visualize entropy distributions across topics and difficulty levels. This visualization method effectively displays not only central tendencies and spread but also highlights data density and outliers in a granular manner. Figure 4: Entropy distributions by topic and difficulty level visualized using boxen plots with swarm overlays. Difficulty level 3 datasets consistently show higher entropy and wider distributions, reflecting greater model uncertainty. The plot reveals several important insights: • Higher Median and Spread at Difficulty Level 3: Across all domains, the median entropy and interquartile ranges for difficulty level 3 datasets are notably larger than those for level 1, consistent with increased uncertainty in model predictions on harder questions. • Domain Variation in Entropy: Biology datasets exhibit relatively lower entropy values overall, which aligns with the statistical tests showing insignificant entropy differences between difficulty levels in this domain. History and Math domains display substantially higher entropy at difficulty level 3, indicating a clear gradation of complexity. • Presence of Outliers and Data Density: The swarm plot overlay reveals the distribution density and presence of multiple outliers with unusually high entropy values, particularly in the higher difficulty datasets. This suggests that a subset of questions poses exceptional uncertainty for the model, possibly due to ambiguous or complex content. • Skewness in Distribution: Especially in the Math-3 dataset, entropy distributions show positive skewness, indicating a heavier tail toward higher uncertainty values, reinforcing the notion of increased challenge in these questions. This visualization therefore provides a nuanced and rich depiction of how difficulty modulates model uncertainty, confirming and extending conclusions drawn from entropy statistics and statistical hypothesis tests. Appendix F Qualitative MCQ Examples In this appendix, we present representative multiple-choice questions generated by our system, illustrating how questions are grounded in the constructed knowledge graph (see Figure 5 for a question generation example). Each example consists of the question’s data and the underlying knowledge route. These examples span diverse domains and are arranged to demonstrate increasing reasoning complexity as the length of the knowledge graph path (number of hops) grows from one to three. These exemplars can clarify on how our system integrate factual triples to formulate coherent questions of varying difficulty, with the correct answer highlighted in bold. Biology – Level 1 Route: "medicine & biology" \xrightarrow{\textsf{INCLUDES}} "Pharmacology" Question: Which field of study is included in the comprehensive overview of medicine and biology, focusing on the effects of drugs on biological systems? A. Toxicology B. Microbiology C. Pharmacology D. Immunology Biology – Level 2 Route: "medicine & biology" \xrightarrow{\textsf{UTILIZES}} "biological insights" \xrightarrow{\textsf{INFORMS}} "medicine" Question: What comprehensive academic overview provides detailed insights into the definition, scope, domains, and applications of both medicine and biology? A. Brief medical summary B. Textbook on pharmacology C. Medicine and Biology Overview D. Public–health article Biology – Level 3 Route: "Biomedicine" \xrightarrow{\textsf{APPLIES}} "biological principles" \xrightarrow{\textsf{ENCOMPASS}} "genetics..." \xrightarrow{\textsf{IS\_A\_BRANCH\_OF}} "biology" Question: Which branch of medical science integrates biological and physiological principles into clinical practice? A. Pharmacology B. Genetics C. Biomedicine D. Immunology History – Level 1 Route: "Social History" \xrightarrow{\textsf{EXAMINES}} "societal structures" Question: Which historical subfield focuses on the experiences and perspectives of ordinary people? A. Economic B. Political C. Cultural D. Social History – Level 2 Route: "context" \xrightarrow{\textsf{IS\_VITAL\_IN}} "archaeology" \xrightarrow{\textsf{FOCUSES\_ON}} "human history" Question: What term refers to the surrounding circumstances that influence the interpretation of human history in archaeology? A. Background B. Setting C. Context D. Environment History – Level 3 Route: "Ottoman Empire" \xrightarrow{\textsf{ENTERED}} "World War I" \xrightarrow{\textsf{ON\_THE\_SIDE\_OF}} "Central Powers" \xrightarrow{\textsf{OPPOSED}} "Allied Powers" Question: Which major empire entered World War I on the side of the Central Powers, opposing the Allied Powers? A. Austro-Hungarian Empire B. German Empire C. Ottoman Empire D. Russian Empire Mathematics – Level 1 Route: "fundamental branch" \xrightarrow{\textsf{REFERS\_TO}} "arithmetic" Question: What fundamental branch of mathematics studies numbers and basic operations? A. Algebra B. Geometry C. Calculus D. Arithmetic Mathematics – Level 2 Route: "linear algebra" \xrightarrow{\textsf{PROVIDES\_TOOLS\_FOR}} "solving systems of linear equations" \xrightarrow{\textsf{USED\_IN}} "optimization" Question: Which branch of mathematics provides essential tools for solving systems of linear equations, a technique frequently used in optimization problems? A. Calculus B. Discrete Mathematics C. Linear Algebra D. Probability Theory Mathematics – Level 3 Route: "eigenvectors" \xrightarrow{\textsf{ARE\_KEY\_CONCEPTS\_IN}} "linear algebra" \xrightarrow{\textsf{IS\_A\_BRANCH\_OF}} "mathematics" \xrightarrow{\textsf{INVOLVES}} "logical reasoning" Question: Which branch of mathematics involves the study of eigenvectors and is essential for logical reasoning in proofs and theorems? A) Geometry B) Linear Algebra C) Calculus D) Statistics Appendix G Graph Update and Curator Mechanism In this appendix, we describe KNIGHT’s Graph Update and Curator Mechanism , which ensures that the knowledge graph grows in a controlled, non-redundant manner. We reference both the illustrative traversal in Figure 5 and the pseudocode of the Curator module in Algorithm 2 . G.1 Formal Definition of Node Curation Let G=(V,E) be the current directed graph, and let t\in V be the node under expansion. The relation extraction stage applied to node t proposes a raw candidate set R=\{\,t^{\prime}_{1},t^{\prime}_{2},\dots\} of potential new child nodes. The Curator filters R to produce a curated subset C\subseteq R of unique, relevant new topics that satisfy the following conditions: \begin{split}C=\bigl\{\,t^{\prime}\in R\mid\;&\forall v\in V,\;\neg\,\mathrm{Equiv}(t^{\prime},v)\\ &\quad\wedge\quad\phi(t^{\prime})=\textsc{True}\bigr\},\end{split} (11) where \mathrm{Equiv}(t^{\prime},v) holds if t^{\prime} is judged semantically equivalent to v , either by exact or normalized string match or by high semantic similarity based on cosine similarity of embeddings. Curator is also a content filter that validates candidate topics by enforcing: 1. Object type agreement: Ensures the candidate’s semantic type aligns with Wikidata taxonomy [ 68 ] . 2. Entailment consistency: Validates logical consistency between the candidate’s description and the relation via natural language inference (NLI) probes [ 48 ] . 3. Content-policy compliance: Checks adherence to content guidelines and filters out hallucinated or inappropriate information [ 58 ] . Only candidates passing all these checks are retained; others are discarded and flagged for human audit by the Curator module. Empirically, this multi-stage filtering prunes approximately 7.6% of candidate edges across domains (§ 4 ), substantially improving the quality and answerability of the generated knowledge graph. G.2 Knowledge-Graph Construction Given a seed topic v_{0} , KNIGHT crawls Wikipedia and populates a property graph G=(V,E) in Neo4j through a recursive expansion process. For each term (starting with v_{0} ), the system generates a comprehensive description. This description generation conditionally utilizes Wikipedia as contextual information if available and deemed relevant by an LLM; otherwise, it relies on the LLM with a structured prompt. From the generated description, subject-predicate-object triplets are extracted. These triplets identify new potential entities (nodes) and their relationships (edges). The newly identified entities are then treated as new terms, and the description generation and relation extraction process is applied recursively to them. The depth parameter d controls the maximum extent of this recursive expansion from the initial seed topic, effectively defining the scope of the resulting KG: V_{d}\;=\;\bigl\{v\in V\;\big|\;{\rm dist}\bigl(v_{0},v\bigr)\leq d\bigr\}, (12) where {\rm dist}(\cdot,\cdot) denotes the shortest-path distance within the constructed graph. Depth therefore acts as an intrinsic hardness knob : increasing d introduces longer reasoning chains, echoing multi-hop QA observations [ 72 ] . G.3 Graph Update and Curator Mechanism In this appendix, we describe KNIGHT’s Graph Update and Curator Mechanism , which ensures that the knowledge graph grows in a controlled, non-redundant manner. We reference both the illustrative traversal in Figure 5 and the pseudocode of the Curator module in Algorithm 2 . G.4 Formal Definition of Node Curation Let G=(V,E) be the current directed graph, and let t\in V be the node under expansion. The relation extraction stage applied to node t proposes a raw candidate set R=\{\,t^{\prime}_{1},t^{\prime}_{2},\dots\} of potential new child nodes. The Curator filters R to produce a curated subset C\subseteq R of unique, relevant new topics that satisfy the following conditions: \begin{split}C=\bigl\{\,t^{\prime}\in R\mid\;&\forall v\in V,\;\neg\,\mathrm{Equiv}(t^{\prime},v)\\ &\quad\wedge\quad\phi(t^{\prime})=\textsc{True}\bigr\},\end{split} (13) where \mathrm{Equiv}(t^{\prime},v) holds if t^{\prime} is judged semantically equivalent to v , either by exact or normalized string match or by high semantic similarity based on cosine similarity of embeddings. Algorithm 2 illustrates the process for detecting duplicates and semantic aliases ( \neg\,\mathrm{Equiv}(t^{\prime},v) ). Curator is also a content filter that validates candidate topics by enforcing: 1. Object type agreement: Ensures the candidate’s semantic type aligns with expected taxonomy. 2. Entailment consistency: Validates logical consistency between the candidate’s description and the relation via natural language inference (NLI) probes. 3. Content-policy compliance: Checks adherence to content guidelines and filters out hallucinated or inappropriate information using our designed RAG system. Only candidates passing all these checks are retained; others are discarded and flagged for human audit by the Curator module. If a semantic alias is detected, the candidate node t^{\prime} is merged with the existing node v , typically by discarding t^{\prime} and re-attributing relations to v . Empirically, this multi-stage filtering prunes approximately 7.6% of candidate edges across domains (§ 4 ), substantially improving the quality and answerability of the generated knowledge graph. Algorithm 2 Curator module: Uniqueness and Alias Filtering Input: current graph G=(V,E) ; topic t ; raw candidates R . Output: curated set C\subseteq R . 1: C\leftarrow\emptyset 2: for \textbf{each }t^{\prime}\in R do 3: s\leftarrow\mathrm{normalize}(t^{\prime}) 4: if s\in\{\mathrm{normalize}(v)\mid v\in V\} then 5: continue \triangleright duplicate name 6: end if 7: for \textbf{each }v\in V do \triangleright semantic alias check 8: if \cos(\mathbf{e}(t^{\prime}),\mathbf{e}(v))\geq\tau then 9: merge t^{\prime} with v \triangleright alias 10: continue to next t^{\prime} 11: end if 12: end for 13: C\leftarrow C\cup\{t^{\prime}\} \triangleright unique and valid 14: end for 15: return C Once the curated set C\subseteq R is determined, the knowledge graph is expanded by adding each new topic t^{\prime}\in C as a vertex and linking it to its parent t . Formally, this update is expressed as: \begin{split}V(G)&\;\leftarrow\;V(G)\;\cup\;C,\\ E(G)&\;\leftarrow\;E(G)\;\cup\;\{(t\to t^{\prime})\mid t^{\prime}\in C\}.\end{split} (14) This procedure guarantees semantic uniqueness and prevents redundancy by merging semantically equivalent nodes, for example, recognizing that “Second World War” and “World War II” represent the same entity. Beyond the detection and merging of duplicates and semantic aliases, the Curator further applies critical validation checks to ensure the integrity and relevance of graph expansions. These checks include verifying that the candidate node’s semantic type conforms to expected classes based on the Wikidata taxonomy [ 68 ] , ensuring that the relations and descriptions are logically consistent through natural language inference (NLI) techniques [ 48 ] , and screening for compliance with content policies to exclude hallucinated or inappropriate information using our designed RAG system. [ 58 ] . G.5 Question Generation Given a topic–seed node v_{0} , KNIGHT samples a length- d path through the knowledge graph: v_{0}\xrightarrow{r_{1}}v_{1}\xrightarrow{r_{2}}\dotsb\xrightarrow{r_{d}}v_{d}, (15) where edges (r_{i}) and intermediate nodes (v_{i}) , along with their descriptions, are retrieved from the graph. The facts along this chain are verbalized into declarative sentences and embedded into a few-shot prompt template. An LLM (by default GPT-4o-mini) then generates a multiple-choice question (MCQ) consisting of (i) a question stem, (ii) one correct answer option, and (iii) three plausible distractors often related to knowledge graph concepts [ 73 ] . To increase item diversity without additional graph traversals, KNIGHT produces both forward and reverse variants of the path in Equation ( 15 ): 1. Forward mode ( \rightarrow ): The answer node is v_{d} , and the question stem is framed from the perspective of the seed node v_{0} , “moving outward.” For example, for d=2 , the question might be: “Which entity, founded by v_{0} , later merged with v_{1} ?” 2. Reverse mode ( \leftarrow ): The path is traversed backwards, treating v_{0} as the correct answer. The stem is phrased around the end node v_{d} , e.g., “ v_{d} traces its origins to which founding entity?” Empirically, reverse questions increase model entropy by 15–20% (Section 4 ), serving as an effective difficulty augmentation while maintaining factual grounding. Increasing the path length d typically leads to progressively harder MCQ templates by requiring the integration of information across more hops in the knowledge graph. All generated MCQs, regardless of direction, are subsequently filtered by rigorous quality criteria. Figure 5 illustrates a sample knowledge-graph traversal starting from the seed node Hafez . At depth 1 (purple path), the algorithm identifies Shiraz as a connected node. Deeper traversals (not shown) discover nodes such as “7th century,” “Iran,” and “>90 million,” which correspond to progressively harder MCQ templates. The Curator module ensures semantic uniqueness by adding entities like “Shiraz” only once, even if discovered through multiple paths. Figure 5: Knowledge-graph traversal from Hafez to Shiraz , 7th century , Iran , and >90 million , generating MCQ templates of increasing difficulty. The purple path denotes a 1-hop (Level 1) question. Appendix H Generation Speed and Experimental Hardware Setup Our entire QA dataset generation pipeline was executed on Google Colab, utilizing an NVIDIA Tesla T4 GPU with 12 CPU cores. Despite this relatively modest hardware setup compared to large-scale compute clusters, the framework demonstrated efficient runtime performance. As shown by the empirical measurements in Subsection 4.4 , KNIGHT enables fast and scalable generation without reliance on expensive or specialized hardware. This efficiency makes it practical for widespread use in research or educational applications, effectively addressing the common time and resource barriers associated with the construction of large-scale QA datasets. Appendix I System Parameters and Configuration The behavior and output quality of the KNIGHT framework are influenced by various parameters controlling the underlying models and processes. These parameters are typically configured before running the generation pipeline and allow for tuning performance, creativity, and filtering thresholds. Here, we provide an outline of key parameters and their roles in our system, along with the empirical considerations that led to our default choices. I.1 Language Model Inference Parameters The Large Language Models used for description synthesis ( \mathcal{L}_{\mathrm{desc}} ), relation extraction ( \mathcal{L}_{\mathrm{rel}} ), MCQ generation ( \mathcal{L}_{\text{q}} ) and validation ( \mathcal{L}_{\text{val}} ) are controlled by the standard generative parameters used primarily. The exact model name is configurable via the environment and key parameters can be configured within the system’s source files including: • Temperature: Temperature controls the randomness of the output distribution during the phase of token generation which ranges from 0.0 to 1.0. With the base of empirical testing across different tasks we found that a moderate temperature is beneficial for creative tasks like initial description generation, while a lower temperature is crucial for structured output tasks like triplet extraction where precision is always the first priority. For initial LLM responses like the description generation task, a moderate temperature of 0.4 is used to balance creative response generation with factual grounding. Initial tests with higher temperatures led to less coherent text, while lower temperatures reduced desired linguistic variation. For triplet extraction, a lower temperature of 0.1 is used to encourage more deterministic and focused extraction of structured information. Testing higher values resulted in less reliable triplet formats, failing to consistently adhere to the required structure. • Max Tokens: This parameter is set to manage output size and prevent excessive generation costs. In our implementation, the triplet extraction process explicitly sets this to 2000 . This value was chosen based on testing that showed this limit is generally sufficient to generate all the information needed in the output, while preventing extremely long, potentially irrelevant outputs that could occur without a limit and increase costs or context window issues. The choice of the specific OPENAI_MODEL (defaulting to gpt-4o-mini-2024-05 ) was driven by empirical evaluation. Comparative testing against larger models like GPT-4 demonstrated that for our specific tasks and prompt engineering, gpt-4o-mini-2024-05 provided a strong balance of accuracy, speed, and significantly lower cost, making it the most practical default for large-scale generation pipelines. I.2 Retrieval-Augmented Generation (RAG) Parameters The RAG components, primarily utilizing the Wikipedia API, fetch information used for description synthesis. Key parameters influencing this process include: • Text Splitting Parameters: When fetching full Wikipedia page content for processing, a recursive text splitter is used to break the text into manageable pieces. The RAG system is configured with a chunk_size of 1000 tokens and chunk_overlap of 100 tokens. These values were found to effectively divide the raw text into segments large enough to retain sufficient context and useful data for the LLM while the overlap helps maintain continuity between chunks. Testing smaller sizes sometimes broke up related facts, while larger sizes could exceed context windows or introduce noise without improving retrieval quality. • Maximum Returned Chunk Length: The routine fetching Wikipedia summaries limits the size of the returned text snippet used as context for node description generation. A default limit of 1000 characters is applied in the code. This value was chosen because empirical observation showed that snippets significantly longer than this rarely added significant value for description synthesis and increased prompt length unnecessarily, potentially diluting the most relevant information for the LLM. Also shorter values deprive the LLM from useful and important context data. • Number of Search Results Checked: When the system needs information about a concept, it searches Wikipedia. This parameter limits the number of top search results from this initial search that the system will look at more closely for relevance before start fetching content from each page which has 5 as the default value. Checking fewer than 5 often missed relevant pages which results in limiting the scope of the generated graph, while checking significantly more added considerable latency and cost without a proportional increase in the retrieval of highly relevant content. These parameters try to retrieve relevant, concise, and contextually appropriate information to ground the generated descriptions and optimize the process both in quality and efficiency. I.3 Graph Construction and Curator Parameters Only one parameter involves in the KG construction and curation process: • Maximum Branches: This parameter controls how many new triplets are extracted from each main idea’s (node’s) description during the recursive graph expansion with 2 as the default value. Through empirical testing, we explored limiting branches to 1, 2, 3 or 4. Limiting to 1 often resulted in a shallow, less interconnected graph structure with missing important related concepts. Allowing more than 2 branches significantly increased computational cost and graph size without consistently yielding a proportional increase in the quality or relevance of generated question paths and sometimes introduced noise. I.4 Validation and Filtering Parameters Parameters governing the validation and filtering of generated QA pairs include: • Validation Sample Rate: This parameter allows specifying a sample rate ranging from 0.0 to 1.0 for LLM-based validation of generated QA pairs. 1.0 is the default used for complete coverage in our experiments reported here to provide the best possible quality, but the parameter was added to enable flexibility for situations where speed or cost is more important than verifying each and every pair generated. By tuning these parameters, users can adjust the trade-off between generation speed, dataset size, linguistic style, and the nuances of difficulty calibration for specific domains and use cases. The default values in our experiments represent a balance to produce high-quality, difficulty-calibrated datasets efficiently.