Difficulty-Controllable Multiple-Choice Question Generation Using Large Language Models and Direct Preference Optimization

Yuto Tomikawa, Masaki Uto · 2025

Read Difficulty-Controllable Multiple-Choice Question Generation Using Large Language Models and Direct Preference Optimization by Yuto Tomikawa, Masaki Uto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19265.txt.

Introduction

Yuto Tomikawa and Masaki Uto build a method that generates multiple-choice reading questions at a chosen difficulty level, aimed at adaptive learning support. Teachers need questions matched to each learner, and earlier generators could control difficulty but could not produce multiple-choice items directly, nor were they trained to hit the target difficulty accurately.

Their approach uses large language models plus direct preference optimization, training the generator explicitly to improve difficulty control. Multiple-choice questions are the most common format in schools, so filling that gap matters. Better control means quizzes that challenge students evenly instead of swinging between trivial and impossible.

Catalog record

CREATORYuto Tomikawa, Masaki Uto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19265

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Difficulty-Controllable Multiple-Choice Question Generation Using Large Language Models and Direct Preference Optimization Yuto Tomikawa, Masaki Uto Abstract Difficulty-controllable question generation for reading comprehension has gained significant attention in the field of education as a fundamental tool for adaptive learning support. Although several neural question generation methods have recently succeeded in controlling difficulty, conventional approaches still face two major limitations. First, they cannot directly generate multiple-choice questions, which are the most widely used question type in educational contexts. Second, they are not explicitly trained to optimize the accuracy of difficulty control, leaving room for further improvement in difficulty controllability. To address these limitations, this study proposes a novel difficulty-controllable multiple-choice question generation method for reading comprehension which leverages a large language model trained using a direct preference optimization technique to improve the accuracy of difficulty control. I Introduction Advances in information technology and the rise of generative artificial intelligence (AI) have made it easier for learners to access a wide range of information of varying quality, highlighting the importance of developing reading comprehension skills to accurately identify important information and understand its content. One common approach to fostering reading comprehension skills is to provide learners with a variety of reading passages and related comprehension questions [ 1 , 2 , 3 , 4 ] . However, manually creating diverse reading comprehension questions for various reading passages presents a significant challenge due to the extensive time and workload required. To address this challenge, automatic question generation (QG), in which questions are generated from reading passages using AI techniques, has attracted significant attention [ 5 , 6 , 7 , 8 , 9 ] . Early QG methods were typically based on rule-based or template-based approaches, relying on manually crafted templates and rules to generate questions [ 1 , 10 , 11 ] . However, creating templates or rules capable of generating diverse, high-quality questions incurs considerable costs. To overcome this issue, QG methods based on deep neural networks that do not require templates or rules have been proposed in recent years [ 12 , 13 , 14 , 15 , 16 ] . While early studies on neural QG have primarily employed recurrent neural networks (RNNs), recent QG methods have used transformer models [ 17 ] . In particular, numerous approaches using pre-trained language models (PLMs) [ 18 , 19 , 20 , 21 ] and their extension into large language models (LLMs) [ 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 ] , both built on transformer architectures, have been proposed [ 32 , 33 , 34 , 13 ] . While these methods enable the generation of high-quality questions, they lack the ability to control the difficulty of the generated questions. When considering the use of QG in the context of learning support and educational assessment, it is desirable to generate questions with controllable difficulty levels. For example, difficulty control in QG could facilitate adaptive question presentation tailored to the ability level of individual learners, thereby enhancing adaptive learning systems [ 35 ] . Furthermore, it would be useful for expanding item pools required for managing large-scale examinations [ 36 ] . Fig. 1: Conceptual diagram of answer-extractive QG (task in previous studies [ 7 ] ) and multiple-choice QG (task in the present study). For these reasons, several difficulty-controllable QG methods have been proposed in recent years [ 37 , 38 , 6 , 39 , 8 , 16 ] . For example, Gao et al. [ 38 ] proposed a method that generates questions with two levels of difficulty: “easy” and “difficult”. In addition, Cheng et al. [ 37 ] proposed a difficulty-controllable QG method that considers the number of sentences that must be referenced in order to reach an answer as the measure of difficulty. However, Tomikawa et al. [ 7 ] pointed out that these methods cannot generate questions with difficulty levels tailored to individual learners’ abilities, and proposed a method that utilizes item response theory (IRT) [ 40 ] , which enables the association between learners’ abilities and question difficulty to be interpreted. Specifically, they developed a method to generate questions across difficulty levels quantified by IRT, proposing a framework for adaptive QG that iteratively estimates learners’ abilities and generates questions with difficulty levels appropriate to those abilities. Although conventional methods have successfully controlled difficulty, they still face two major limitations. 1. These methods aim to generate questions with extractive answers, a question type that requires identifying word spans as answers from a given reading passage, as shown on the left side of Figure 1 . Consequently, they cannot directly generate multiple-choice questions, which are widely used in educational settings. 2. These methods typically train neural QG models by maximizing the conditional likelihood of each question text given a difficulty level and a reading passage. However, they do not explicitly optimize the accuracy of difficulty control, leaving room for further improvement in controllability. To overcome these limitations, in this study, we develop a difficulty-controllable multiple-choice QG method using an LLM with a training approach that explicitly optimizes difficulty control accuracy through direct preference optimization (DPO) [ 28 ] . We first propose a method for generating both question texts and options for multiple-choice questions from a reading passage and a specified difficulty level, as shown on the right side of Figure 1 . This method is implemented using Llama 3.1 [ 41 ] , an open-source LLM. Next, we introduce a method for maximizing difficulty control accuracy based on DPO, a technique derived from reinforcement learning from human feedback (RLHF) [ 23 ] , which aligns the output of LLMs with specific objectives. While conventional methods are trained with supervised fine-tuning (SFT) to maximize the conditional likelihood of each question text given a difficulty level, the proposed method directly optimizes difficulty control accuracy after conventional SFT training via the DPO approach, thereby improving the control accuracy. The proposed method requires a dataset consisting of quadruples of (reading passage, question text, options, question difficulty) for model training, although typical multiple-choice question datasets do not include difficulty levels. To address this, we employ a method for constructing a multiple-choice question dataset incorporating difficulty levels, following the difficulty estimation method proposed by Tomikawa et al. [ 7 ] , which uses IRT [ 42 , 40 , 43 ] and question answering (QA) systems. This study evaluates the effectiveness of the proposed method through experiments using the RACE dataset [ 44 ] , which is widely used in the task of automatic multiple-choice reading comprehension QG. II Related Works In this section, we provide an overview of conventional neural QG methods and difficulty-controllable QG methods. II-A Neural QG for Reading Comprehension Early neural QG methods for reading comprehension have primarily used RNN-based models [ 5 , 45 , 9 ] . For example, Du et al. [ 5 ] proposed a method that generates question text by inputting a reading passage and answer into an RNN encoder, with the obtained output feature vector subsequently fed into an RNN decoder to sequentially generate the question text. Zhou et al. [ 9 ] proposed an extended RNN-based QG method

Read the complete text