Elementary Math Word Problem Generation using Large Language Models

*Nimesh Ariyarathne, *Harshani Bandara, *Yasith Heshan, *Omega Gamage, *Surangika Ranathunga, Dilan Nayanajith, Yutharsan Sivapalan, Gayathri Lihinikaduarachchi, Tharoosha Vihidun, Meenambika Chandirakumar, Sanujen Premakumar, Sanjula Gathsara · 2025

Read Elementary Math Word Problem Generation using Large Language Models by *Nimesh Ariyarathne, *Harshani Bandara, *Yasith Heshan, *Omega Gamage, *Surangika Ranathunga, Dilan Nayanajith, Yutharsan Sivapalan, Gayathri Lihinikaduarachchi, Tharoosha Vihidun, Meenambika Chandirakumar, Sanujen Premakumar, Sanjula Gathsara (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05950.txt.

Introduction

Nimesh Ariyarathne and a large team of co-authors build MathWiz, a system that writes grade school math word problems with large language models. Their paper from two thousand twenty-five notes that math feels hard to many students and failure rates are high. Practice problems help, but teachers spend much time writing good ones by hand.

MathWiz needs just three inputs: how many problems, the grade, and the type, such as adding or taking away. The team tested many models, prompt styles, and ways to add variety and human feedback. Judges and auto checks rated the problems high in quality, with few spelling slips. Models still strayed from the set grade and type at times.

Catalog record

CREATOR*Nimesh Ariyarathne, *Harshani Bandara, *Yasith Heshan, *Omega Gamage, *Surangika Ranathunga, Dilan Nayanajith, Yutharsan Sivapalan, Gayathri Lihinikaduarachchi, Tharoosha Vihidun, Meenambika Chandirakumar, Sanujen Premakumar, Sanjula Gathsara
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05950

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Elementary Math Word Problem Generation using Large Language Models *Nimesh Ariyarathne, *Harshani Bandara, *Yasith Heshan, *Omega Gamage, *Surangika Ranathunga, Dilan Nayanajith, Yutharsan Sivapalan, Gayathri Lihinikaduarachchi, Tharoosha Vihidun, Meenambika Chandirakumar, Sanujen Premakumar, Sanjula Gathsara Abstract Mathematics is often perceived as a complex subject by students, leading to high failure rates in exams. To improve Mathematics skills, it is important to provide sample questions for students to practice problem-solving. Manually creating Math Word Problems (MWPs) is time consuming for tutors, because they have to type in natural language while adhering to grammar and spelling rules of the language. Existing Deep Learning techniques for MWP generation either require a tutor to provide the initial portion of the MWP, and/or additional information such as an equation. In this paper, we present an MWP generation system based on Large Language Models (LLMs) that overcome the need for additional input - the only input to our system is the number of MWPs needed, the grade and the type of question (e.g. addition, subtraction). Unlike the existing LLM-based solutions for MWP generation, we carried out an extensive set of experiments involving different LLMs, prompting strategies, techniques to improve the diversity of questions, as well as techniques that employ human feedback to improve LLM performance. Human and automated evaluations confirmed that the generated MWPs are high in quality, with minimal spelling and grammar issues. However, LLMs still struggle to generate questions that adhere to the specified grade and question type requirements. 1 Introduction Mathematics is often perceived as a complex subject by students, leading to high failure rates in exams ( Casinillo 2019 ) . To improve Mathematics skills, it is important to provide sample questions for students to practice problem-solving. This requires tutors to come up with a diverse set of Mathematics questions manually or by using automated mechanisms. Math word problems (MWPs) are an important type of educational resource that helps assess and improve students’ proficiency in various mathematical concepts and skills ( Wang et al. 2021 ) . An MWP is a mathematical problem expressed in natural language and it requires problem-solving, as well as language comprehension ability (see Figure 1 for example MWPs). However, given that an MWP contains a higher portion of natural language text, creating customized MWPs can be a time-consuming task for the tutors. A viable alternative is to automatically generate MWPs. Deep Learning based systems have shown promising results in this front. Early Deep Learning based works for MWP generation require the user to provide the equation, starting portion (seed) of the MWP, or some context ( Wang et al. 2021 ; Niyarepola et al. 2022 ) . However, these methods entail limitations, particularly in the necessity to accurately define both a seed, equation and/or context that suits the MWP, which can be time-consuming. Furthermore, such methods may not guarantee the diversity of questions, nor they could cater to the requirement of generating questions while adhering to a particular grade or a question type. In this context, generative AI techniques backed by Large Language Models (LLMs) provide a promising avenue. LLMs have created a paradigm shift in the field of Generative AI, with successful applications in many domains ( Gozalo-Brizuela and Merchan 2024 ) . However, for MWP generation, we are only aware of the work of Christ et al. 2024 ; Hwang and Utami 2024 ; Xie et al. 2024 . Xie et al. 2024 used LLMs to modify the values of a given MWP to derive a new MWP, but did not generate MWPs from scratch. Hwang and Utami 2024 simply prompted chatGPT to generate MWPs, but did not fine-tune the chatGPT model further. Christ et al. 2024 fine-tuned the Llama-2 model for MWP generation. Their model was able to generate questions at appropriate reading levels, but could not consider the question type. Their error analysis only considered three error types. Moreover, their model did not guarantee the diversity of the generated MWPs, nor did they go beyond basic fine-tuning. We present an MWP generation system based on LLMs, which is focused on generating MWPs belonging to a user-specified grade and a question type. We carried out an extensive study to identify the best open-source LLM and prompt template for MWP generation. The most promising LLM was further fine-tuned with an MWP dataset that we manually created. We further improved the quality of the generated MWPs by incorporating human feedback to the training process. We also investigated on the hyper-parameters related to MWP generation, and determined the optimal paramter combination to guarantee the diversity of the generated questions, and incorporated a secondary LLM to guarantee the generated MWPs are solvable. There are 6 red socks and 2 blue socks. How many socks are there?(Grade 1, Single-digit Addition) Liam had 40 marbles, lost 15, then won 20 in a game. How many marbles does Liam have now? (Grade2, 2 - Digit Addition & Subtraction) Figure 1: Examples of MWPs (According to the United States Common Core Mathematics Curriculum) In order to evaluate our MWP generation model, we carried out an extensive evaluation involving humans and an LLM-based evaluation technique, where the MWPs generated by the LLM after each improvement was evaluated. Human evaluation was conducted following the participatory research approach 1 1 1 The participants to the research becomes co-authors of the publication. This is increasingly becoming a common approach when a research does not have funds to compensate human participants Nekoto et al. 2020 . . Our evaluation considered 12 error types, as opposed to the three error categories considered by Christ et al. 2024 . The human + LLM evaluation produced a dataset of 4K MWPs with error annotation. To the best of our knowledge, this is the only comprehensively error annotated English MWP corpus. This corpus, as well as the training MWP corpus that we manually created are publicly released under the Mathwizards dataset . 2 Background and Related Work In this section, we discuss the past work related to MWP generation. We also introduce the techniques and concepts related to LLM-based text generation, which were used in this research. 2.1 MWP Generation Early approaches for MWP generation include question re-writing, template-based generation and generation with Neural Networks (NNs) ( Niyarepola et al. 2022 ) . Early NN approaches include training Deep Learning models such as Recurrent Neural Networks (RNNs) ( Medsker et al. 2001 ; Liyanage and Ranathunga 2020 ) , building custom NN models ( Qin et al. 2023 ) , and fine-tuning pre-trained language models such as mT5 ( Xue et al. 2020 ) or GPT ( Liu et al. 2023a ) for the task of MWP generation. In order to generate an MWP from such NN approaches, some form of input has to be provided to the model. This input can be in the form of an equation, context and/or a seed ( Wang et al. 2021 ; Gamage et al. 2024 ; Niyarepola et al. 2022 ; Zhou et al. 2023 ) . However, these generated questions have general errors such as grammatical and spelling errors, as well as Mathematics-specific errors such as having wrong units, unsolvable questions and resulting in unrealistic answers ( Gamage et al. 2024 ) . Moreover,

Read the complete text