Automatic Generation of Inference Making Questions for Reading Comprehension Assessments
Wanjing Anya Ma, Michael Flor, Zuowei Wang · 2025
Read Automatic Generation of Inference Making Questions for Reading Comprehension Assessments by Wanjing Anya Ma, Michael Flor, Zuowei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08260.txt.
Introduction
Wanjing Anya Ma, Michael Flor, and Zuowei Wang test machine generation of inference questions for reading tests, in work from 2025. Inference questions probe whether students connect ideas across a passage. Writing good ones by hand is slow, so test programs want machine help.
The team first built a taxonomy of inference types and audited an item bank against it. A large model then drafted bridging-inference items for set passages, with and without step-by-step prompting. Raters scored item quality, type fit, and the model's reasoning. Agreement between raters ran high across all three scores. Most drafted items passed as usable for grades three to twelve. Type accuracy lagged, though, so human review stays in the loop.
Catalog record
| CREATOR | Wanjing Anya Ma, Michael Flor, Zuowei Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08260
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automatic Generation of Inference Making Questions for Reading Comprehension Assessments Thanks: Work done while at ETS Research Institute Wanjing Anya Ma, Michael Flor, Zuowei Wang Abstract Inference making is an essential but complex skill in reading comprehension (RC). Some inferences require resolving references across sentences, and some rely on using prior knowledge to fill in the detail that is not explicitly written in the text. Diagnostic RC questions can help educators provide more effective and targeted reading instruction and interventions for school-age students. We introduce a taxonomy of inference types for RC and use it to analyze the distribution of items within a diagnostic RC item bank. Next, we present experiments using GPT-4o to generate bridging-inference RC items for given reading passages via few-shot prompting, comparing conditions with and without chain-of-thought prompts. Generated items were evaluated on three aspects: overall item quality, appropriate inference type, and LLM reasoning, achieving high inter-rater agreements above 0.90. Our results show that GPT-4o produced 93.8% good-quality questions suitable for operational use in grade 3-12 contexts; however, only 42.6% of the generated questions accurately matched the targeted inference type. We conclude that combining automatic item generation with human judgment offers a promising path toward scalable, high-quality diagnostic RC assessments. 1 Introduction Figure 1: Overview of automatic item generation and human evaluation. We use GPT-4o to generate bridging-inference RC items for given reading passages via few-shot prompting, comparing conditions with and without chain-of-thought prompts. We prompt each inference type separately: pronominal bridging, text-connecting, and gap-filling inferences. Human evaluation focuses on general item quality, inference type appropriateness, and LLM rationales. Inference-making is an essential yet cognitively demanding skill in reading comprehension (RC) ( O’Brien et al. 2015 ; Kintsch 1998 ) . Inferences are necessary for establishing both local and global coherence within the mental representation of a text ( Graesser et al. 1994 ) . Local inferences connect information across sentences using cohesive devices such as anaphors or category exemplars—for example, in "Bette gulped down the drink. The cold water was very refreshing," the reader infers that the drink refers to cold water ( Cain 2022 , p. 307) . Global inferences, on the other hand, rely on the reader’s prior knowledge to fill in missing details required to make sense of the text —for example, in "The campfire started to burn uncontrollably. Tom grabbed a bucket of water" ( Bowyer-Crane and Snowling 2005 , p. 192) , the reader infers that Tom intended to put out the fire, based on the knowledge that water extinguishes fire. While skilled readers often generate inferences automatically as they engage with text ( Thurlow and van den Broek 1997 ) , children who struggle with comprehension frequently have difficulty constructing these inferences ( Cain et al. 2001 ) . Providing diagnostic information about specific types of inference-making deficits that hinder comprehension can empower educators to provide more effective and targeted reading instruction and intervention ( Bowyer-Crane and Snowling 2005 ; Bayat and Çetinkaya 2020 ) . To achieve this, we need RC assessments that specifically target inference-making types. At the same time, we want to develop scalable item generation methods to enable multi-time testing, monitoring reading development over time. Previous work has demonstrated the ability of large language models (LLMs) to generate effective RC questions ( Uto et al. 2023 ; Säuberli and Clematide 2024 ) . However, whether LLMs can reliably produce questions that target specific inference types remains unclear. Our research is grounded in a real-world diagnostic assessment of reading skills for students in grades 3 through 12 ( Sabatini et al. 2019 ) . The assessment was originally developed at ETS and recently commercialized as ReadBasix. It leverages the science of reading to assess foundational reading skills, such as word recognition and decoding, as well as more complex ones such as RC. In the RC subtest, a student will usually read 4 expository passages and answer multiple-choice questions associated with the passages. The subtest takes about 30 minutes to complete. Like any large-scale reading assessment, there is an ongoing need for more items. To address this demand, we aim to leverage automatic item generation to create new items based on curated passages, and evaluate the quality of these items before collecting student performance data to make them operational. For the purpose of automatic item generation, as illustrated in Figure 1 , we first conducted a literature review on inference-making in the reading comprehension and natural language processing (NLP) text comprehension literature. We developed a taxonomy of inference-making questions, with a focus on bridging inference. We validated this taxonomy by annotating an operational item bank of expert-written RC questions, confirming bridging inference as an important and widely covered sub-construct. Next, we curated six expository passages and manually wrote multiple-choice RC questions for each inference type based on our taxonomy. These examples were then used to prompt GPT-4o ( Hurst et al. 2024 ) via few-shot prompting to generate bridging-inference questions for new reading passages, comparing conditions with and without chain-of-thought (CoT) prompting ( Wei et al. 2022 ) . Finally, three human experts evaluated the quality of the generated questions along three dimensions: overall item quality 1 1 1 The evaluation of overall item quality does not include whether an item is of the required inference type, which is an extra-evaluation. See Table 2 for more details. , appropriate inference type, and whether GPT-4o provided satisfactory reasoning for generating the question. Our results show that LLMs can produce 93.8% good-quality questions suitable for operational use in grade 3-12 contexts; however, only 42.6% of the generated questions accurately match the targeted inference type. Nevertheless, the overall coverage of inference types closely mirrors what we observe in our operational item bank. We conclude that combining automatic item generation with human judgment offers a promising path toward scalable, high-quality diagnostic RC assessments. In summary, we make the following contributions in this paper: 1. We develop and validate a taxonomy for inference-making questions used in multiple-choice RC assessments, and demonstrate its its value for future item development. 2. We introduce a novel NLP task where language models generate RC questions targeting specific inference types, providing a new way to assess their reasoning abilities. The training item bank will be released for replication and benchmarking. 3. We demonstrate GPT-4o’s potential in generating RC questions for operational use and its limitations in accurately generating specific types of inference questions. 2 Related Work 2.1 Question generation for reading comprehension assessments Automatic question generation is a well-established task in NLP, especially within educational applications, to reduce the high costs of manual question authoring and to ensure a steady supply of new, high-quality items ( Kurdi et al. 2020 ) . Early approaches rely on rule-based or template-based methods ( Araki et al. 2016 ; Flor and Riordan 2018 ) , as well as the use of discourse connectives to generate questions ( Agarwal et al. 2011 ) . Later approaches extensively used neural systems for question generation ( Mulla and Gharpure 2023 ) . More recent work demonstrates that LLMs hold promise in generating high-quality RC questions, using techniques such as fine-tuning ( Uto et al. 2023 ; Perkoff et al. 2023 ; Ghanem et