A Multi-Agent Approach to Validate and Refine LLM-Generated Personalized Math Problems
Fareya Ikram, Nischal Ashok Kumar, Junyang Lu, Hunter McNichols, Candace Walkington, Neil Heffernan, Andrew S. Lan · 2026
Read A Multi-Agent Approach to Validate and Refine LLM-Generated Personalized Math Problems by Fareya Ikram, Nischal Ashok Kumar, Junyang Lu, Hunter McNichols, Candace Walkington, Neil Heffernan, Andrew S. Lan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05160.txt.
Introduction
Ikram and six colleagues build a system that tailors math problems through rounds of generate, check, and revise. Language models can tailor problems, but theirs often have wrong numbers, hard reading, weak fit, and math slips. Four checker agents test whether problems can be solved, look real, read well, and fit student interests. On 600 ASSISTments problems, tailored to 20 topics, one round of revision cuts realism and fit failures, the most common first-round failures. Checkers score best on realism and worst on fit.
Catalog record
| CREATOR | Fareya Ikram, Nischal Ashok Kumar, Junyang Lu, Hunter McNichols, Candace Walkington, Neil Heffernan, Andrew S. Lan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34384 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05160
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Multi-Agent Approach to Validate and Refine LLM-Generated Personalized Math Problems Fareya Ikram, Nischal Ashok Kumar, Junyang Lu, Hunter McNichols, Candace Walkington, Neil Heffernan, Andrew S. Lan Abstract Students benefit from math problems contextualized to their interests. Large language models (LLMs) offer promise for efficient personalization at scale. However, LLM-generated personalized problems may often have problems such as unrealistic quantities and contexts, poor readability, limited authenticity with respect to students’ experiences, and occasional mathematical inconsistencies. To alleviate these problems, we propose a multi-agent framework that formalizes personalization as an iterative generate–validate–revise process; we use four specialized validator agents targeting the criteria of solvability, realism, readability, and authenticity, respectively. We evaluate our framework on 600 problems drawn from a popular online mathematics homework platform, ASSISTments, personalizing each problem to a fixed set of 20 student interest topics. We compare three refinement strategies that differ in how validation feedback is coordinated into revisions. Results show that authenticity and realism are the most frequent failure modes in initial LLM-personalized problems, but that a single refinement iteration substantially reduces these failures. We further find that different refinement strategies have different strengths on different criteria. We also assess validator reliability via human evaluation. Results show that reliability is highest on realism and lowest on authenticity, highlighting the need for better evaluation protocols that consider teachers’ and students’ personal characteristics. 1 Introduction Context personalization, tailoring instructional content to align with students’ individual interests, has been shown to increase engagement and learning outcomes across diverse educational settings [ 13 ] . This task has long been constrained by the effort required for teachers and curriculum developers. The rise of large language models (LLMs) offers potential for educational tasks to be personalized to fine-grained student interests at scale [ 1 ] . However, research cautions that personalization will not effectively facilitate learning unless it connects to students’ authentic real-world experiences [ 23 ] . Current LLM-based personalization tools face several technical challenges that may limit their adoption in classroom settings. Studies of teacher interactions with these systems identify four recurring challenges. First, personalized problems may contain mathematical inaccuracies, such as invalid solutions, inconsistent quantities, or ambiguous setups [ 3 ] [ 6 ] . Second, these problems may include unrealistic quantities or scenarios that violate real-world norms [ 22 ] [ 24 ] . Third, the problems often lack authenticity, failing to align with students’ actual interests or lived experiences [ 22 ] . Finally, research suggests LLM-generated content may increase reading burdens on students [ 4 ] [ 17 ] . In order to address the challenge of balancing multiple constraints, researchers have explored iterative refinement and structured control of agents [ 9 ] . Iterative refinement strategies like Self-Refine [ 14 ] and Reflexion [ 20 ] demonstrate that LLMs can improve outputs through critique-and-revise loops, relying on undifferentiated self-feedback that may overlook domain-specific constraints. We address the multi-faceted nature of context personalization by proposing a multi-agent approach that employs specialized validation and refinement agents for personalizing math problems. Recent work in the field of generative AI has demonstrated that multi-component agentic LLM pipelines outperform single-prompt approaches on complex generation tasks with multi-dimensional criteria [ 2 ] . However, a key question remains on how validation feedback, especially for the task of context personalization, should be coordinated into effective revisions. For example, two criteria, such as adding realistic contextual details while maintaining appropriate readability, need to be balanced. In this work, we experiment with a multi-agent framework that employs iterative generate-validate-revise cycles with four specialized validator agents, i.e., agents targeting each of the identified challenges. We compare three refinement strategies: (1) Centralized Refinement, which aggregates feedback from all validator agents before revision; (2) Centralized Refinement with Planning, which adds a planning component to structure and prioritize feedback; and (3) Decentralized Refinement, which couples each validator agent with a dedicated refinement agent. We evaluate our framework by first comparing the performance of three different refinement strategies. Second, we assess the reliability of our validator agents via human evaluation. Our results show that realism and authenticity are the most common failure modes in initial LLM-personalized problems, but that a single refinement iteration substantially reduces these errors. We further find that refinement behavior is criterion-dependent: Decentralized Refinement reduces realism and readability failures more quickly, while Centralized Refinement with Planning converges fastest on authenticity. Finally, our human evaluation shows that the realism validator agent matches human judgments more reliably than the authenticity validator agent, suggesting that authenticity assessments often depend on an individual’s familiarity with the topic. Our results motivate future evaluations grounded in teacher and student judgments. 2 Related Works Obstacles in LLM Generated Educational Content Research on LLM-based educational personalization has documented specific patterns of concern. Several K-12 teacher-facing generative AI products, including Khanmigo [ 12 ] and MagicSchool [ 15 ] , provide tools for adapting lessons to students’ interests, yet teachers report these adaptations as insufficient. Research in mathematics education finds that typical mathematical tasks are often framed as contrived and unrealistic with respect to students’ interests, and inauthentic with respect to students’ actual experiences with quantities and measurement [ 18 ] . Studies of teacher-LLM interactions reveal that teachers frequently prompt systems to adjust unrealistic problem formulations where numbers violate real-world norms, for example, a bottle of water costing 100 dollars [ 22 ] [ 24 ] . Teachers also attend closely to popular culture references, noting when references like Fortnite resonate more with their students than alternatives like Roblox, demonstrating concern for contextual authenticity and timeliness. Additionally, teachers adjust readability levels of LLM output to meet their students’ needs [ 22 ] . While mathematical accuracy has improved since earlier investigation, contemporary models still generate problems with mathematical errors [ 3 ] . Multi-Agent Architectures Multi-agent systems have emerged as a promising paradigm for educational content generation, demonstrating that complex educational tasks benefit from specialized, coordinated agents rather than single models. For instance, EduAgentQG [ 10 ] demonstrates that multi-agent workflows can generate diverse, high-quality problems aligned to educational objectives through specialized agents for planning, writing, and validation. While such approaches advance question generation, creating new problems from scratch to cover learning objectives, our work addresses a fundamentally different and complementary challenge: context personalization. Rather than generating new problems, we adapt existing, validated problems to student interests while preserving their underlying mathematical structure and pedagogical intent. Similarly, recent work on multi-agent collaborative frameworks for math problem generation [ 11 ] employs teacher-critic cycles where a teacher agent generates question-answer pairs and a generic critic provides iterative feedback on clarity, relevance, and difficulty alignment. Their evaluation reveals incremental improvements over baseline approaches, with the greatest gains observed in difficulty matching and relevance, highlighting the value of multi-agent refinement for educational content generation. However, their approach employs a single generic critic to assess all quality dimensions at once, which may limit the system’s ability to provide targeted, criterion-specific feedback to guide revisions. The value of multi-agent approaches extends beyond question generation to other educational content tasks requiring multi-dimensional quality assessment. Yang et al. [ 26 ] demonstrate this in teacher professional development, where their LLMAgent-CK framework employs multiple agents in collaborative discussion to assess teachers’ mathematical