Towards Just-in-Time Adaptive Feedback: Enhancing Student Learning via Knowledge-Grounded LLM
Younghun Lee, Amir Bralin, Nobel Sanjay Rebello, Dan Goldwasser · 2026
Read Towards Just-in-Time Adaptive Feedback: Enhancing Student Learning via Knowledge-Grounded LLM by Younghun Lee, Amir Bralin, Nobel Sanjay Rebello, Dan Goldwasser (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26405.txt.
Introduction
Younghun Lee and three colleagues set out a way to give students timely, tailored feedback while they still work. Their system reads strategy essays, spots likely error types in the reasoning, and sends quiet prompts that clear up missing or wrong ideas. Prompts draw on expert subject knowledge. That keeps the language model tied to course content. Run in a course of over a thousand students, it lifted scores by more than 80 percent against prior terms.
Catalog record
| CREATOR | Younghun Lee, Amir Bralin, Nobel Sanjay Rebello, Dan Goldwasser |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41709 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26405
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Just-in-Time Adaptive Feedback: Enhancing Student Learning via Knowledge-Grounded LLM Younghun Lee, Amir Bralin, Nobel Sanjay Rebello, Dan Goldwasser Abstract Educational interventions are effective tools for enhancing student learning. While Large Language Models (LLMs) allow for generating adaptive feedback at scale, current studies lack clear methodologies for providing Just-in-Time (JiT) feedback in authentic instructional settings. In this paper, we present a framework that provides adaptive feedback by grounding LLMs with domain-specific expert knowledge. Our approach collects written reasoning logic (strategy essays) from students, analyzes potential error types based on the content of that reasoning, and delivers non-intrusive feedback designed to clarify missing or incorrect concepts. We deploy this framework in a large-scale university course ( N > 1,000 N>1,000 ), where it improved student performance by over 80% compared to previous semesters. Lastly, we validate the framework’s pedagogical utility by analyzing the learning trajectories; we demonstrate how iterative conversations with LLM facilitate shifting one’s misconception to correct understanding. Figure 1: Overall framework of our Just-in-Time adaptive feedback LLM. In the first phase, we obtain domain knowledge from human experts regarding the appropriate feedback for students’ strategy essays and the types of errors they made. At deployment, we ground the LLM with this knowledge and provide adaptive feedback to the students based on their written strategy essays. 1 Introduction In STEM education, students are often observed to engage in “Recursive Plug-and-Chug”—also known as “Formula Hunting”—where the goal is simply to fill a slot in a formula rather than to make sense of the problem ( 5 ; 32 ) . Existing studies indicate that while experts analyze problems based on their deep logical structure and principles, novices are often primed to surface features, leading to unsuccessful problem-solving approaches. ( 21 ) . Strategy writing has emerged as a powerful intervention to tackle this problem. Research has shown that asking students to articulate their strategies for solving problems can improve their problem-solving skills ( 20 ) . With a combination of argumentation and appropriate prompts, students focus more on deep structure rather than the surface features of the problem and starts solving the problem with conceptual analysis, avoiding novice, unproductive strategies ( 8 ; 21 ; 7 ; 28 ) . Despite the benefits, strategy writing is difficult to implement at scale. The primary impediment is the feedback bottleneck. Feedback is most effective when it is integrated into the learning process through formative assessment ( 13 ) , and provided prior to completion ( 14 ) . However, providing real-time, formative feedback to thousands of students is prohibitively time-consuming for human instructors. Thus, the true potential of strategy writing to enhance student learning remains largely unrealized. The advance of generative Large Language Models (LLMs) offers a promising pathway toward overcoming the scalability barrier. Recent work proposes adaptive scaffolding frameworks that utilizes LLMs to help students understand course materials ( 31 ; 19 ; 17 ; 38 ) . However, they lack exploration of intervention-oriented interactions as well as grounding the models with domain-specific knowledge. In this paper, we address these challenges by exploring dimensions of adaptive feedback and implementing a framework that provides Just-in-Time interventions using knowledge-grounded LLMs. First, we collect strategy essays from students over a couple of semesters and obtain historical data that connect the essays to the types of errors students had made. Human experts then annotate feedback on a small portion of the data considering the content of the essays, as well as the error types. Using these annotated instances as few-shot examples, the LLM identifies misconceptions in the strategy essays and provide real-time, adaptive feedback. Rather than providing a ‘shortcut’ to the correct answer, the feedback directs the student’s attention to the underlying concepts and their reasoning logic (see Figure 4 and Figure 7 for examples). This ensures the intervention disrupts the ‘formula-hunting’ heuristic by letting students reflect on the problem’s deep structure. Figure 1 illustrates the overall framework we propose. We experiment our framework on a large-scale university physics course ( N>1,000 ). Experimental results show that the overall student performance is improved when the feedback framework is deployed, by over 80% compared to previous semesters. Based on further analyses on self-reported survey and conversational instances, we argue that the framework effectively facilitates a shift from initial misconceptions to the correct understanding. 2 Related Work Educational research has shown that feedback is one of the important drivers of learning 36 ; 9 . Feedback can facilitate improvements in learners’ understanding and skills 14 by narrowing the gap between actual and desired performance 4 . The effectiveness of feedback increases with the information that it contains. Previous research 18 has also shown that moderators such as timing, specificity, and task complexity affect how learners receive and use feedback 12 ; 3 . Feedback is effective if it is sufficiently detailed 27 , usable 35 , and facilitates change 29 , such that learners can test their new understandings 26 . In asynchronous and isolated online settings 22 , interactive dialogues can be especially useful 37 as students cannot easily interact with their peers 10 which put significant weight on the feedback comments they receive 23 . Recent research on educational LLM applications has shifted from general assistance to specialized interventions. For instance, 25 used GPT-4 as a teacher to provide non-intrusive hints but relied on LLM-simulated student agents using a weaker LLM (GPT-3.5), rather than applying it to real classroom settings. In contrast, 6 and 15 designed LLM feedback to apply to students, but their evaluation metrics focused on student-reported helpfulness survey and qualitative analysis of feedback content, rather than measuring actual improvements in student performance. More recent studies have shown LLM-generated feedback can help improve student performance in class. 11 used an LLM-based Socratic chatbot to scaffold expert-like reasoning. 38 implemented a persona-driven multi-agent dialogue system, showing the student’s performance in problem-solving improves as they interact more with LLMs. 17 integrated an AI tutor in a college Physics course and compared student’s performance on a quiz between an AI tutor and in-class lessons. The results show that when the model is applied to a class size of 200, the AI tutor almost doubled the learning gains compared to in-class lessons. In this paper, we introduce three novel contributions. Unlike reactive chatbots that are susceptible to formula-hunting behaviors of the students, our system proactively analyzes the reasoning logic within the strategy essays that guides students to focus on deep structures in solving quiz problems. Additionally, we ground LLMs with domain-specific knowledge that is carefully annotated by human experts, which brings significant benefits compared to persona-based prompt engineering. Lastly, we provide a robust, automated intervention that remains pedagogically sound at a massive scale ( N>1,000 ). 3 Problem Formulation 3.1 Instructional Setting Our research context is a large enrollment calculus-based physics course for engineers and physical scientists at a large U.S. Midwestern land grant university. The focus of the course is the development of problem-solving skills focused on applying key physics principles pertaining to mechanics across three units: Newton’s Laws and linear momentum, work and energy, and angular momentum. The annual enrollment of the course is around 3,300 students (1,500 in fall, 1,800 in spring), about 25% women,