How Adding Metacognitive Requirements in Support of AI Feedback in Practice Exams Transforms Student Learning Behaviors Conference: Proceedings of the Twelfth ACM Conference on Learning @ Scale; July 21–23, 2025; Palermo, Italy Proceedings of the Twelfth ACM Conference on Learning @ Scale (L@S ’25),

Mak Ahmad, Prerna Ravi, David Karger, Marc Facciotti · 2025

Read How Adding Metacognitive Requirements in Support of AI Feedback in Practice Exams Transforms Student Learning Behaviors Conference: Proceedings of the Twelfth ACM Conference on Learning @ Scale; July 21–23, 2025; Palermo, Italy Proceedings of the Twelfth ACM Conference on Learning @ Scale (L@S ’25), by Mak Ahmad, Prerna Ravi, David Karger, Marc Facciotti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13381.txt.

Introduction

Mak Ahmad, Prerna Ravi, David Karger, and Marc Facciotti test a practice exam system that pairs AI feedback with required self-reflection. Giving detailed, personal feedback in huge undergraduate science classes is a standing problem. Their system, used in a large introductory biology course, asks students to explain each answer and rate their confidence, then has GPT-4o respond with tailored feedback and pointers to textbook sections.

Logs covered over twenty-eight thousand question attempts across three midterms, plus surveys and interviews. Feedback style alone made no clear difference to scores. The reflection requirements did the heavy lifting: students said the habits carried into real exams, and about forty percent followed the textbook pointers, far above normal reading rates. Satisfaction ran high, and most students reported stronger confidence and better recall on practiced topics. The authors argue that structured reflection, more than clever feedback, changes how students learn.

Catalog record

CREATORMak Ahmad, Prerna Ravi, David Karger, Marc Facciotti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13381

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Adding Metacognitive Requirements in Support of AI Feedback in Practice Exams Transforms Student Learning Behaviors Conference: Proceedings of the Twelfth ACM Conference on Learning @ Scale; July 21–23, 2025; Palermo, Italy Proceedings of the Twelfth ACM Conference on Learning @ Scale (L@S ’25), July 21–23, 2025, Palermo, Italy DOI: 10.1145/3698205.3729542 ISBN: 979-8-4007-1291-3/2025/07 CCS: Human-centered computing Empirical studies in HCI CCS: Computing methodologies Artificial intelligence CCS: Applied computing Interactive learning environments CCS: Applied computing Computer-assisted instruction CCS: Social and professional topics Student assessment Mak Ahmad, Prerna Ravi, David Karger, Marc Facciotti Abstract . Providing personalized, detailed feedback at scale in large undergraduate STEM courses remains a persistent challenge. We present an empirically evaluated practice exam system that integrates AI generated feedback with targeted textbook references, deployed in a large introductory biology course. Our system specifically aims to encourage metacognitive behavior by asking students to explain their answers and declare their confidence . It uses OpenAI’s GPT-4o to generate personalized feedback based on this information, while directing them to relevant textbook sections. Through detailed interaction logs from consenting participants across three midterms (541, 342, and 413 students respectively), totaling 28,313 question-student interactions across 146 learning objectives, along with 279 post-exam surveys and 23 semi-structured interviews, we examined the system’s impact on learning outcomes and student engagement. Analysis showed that across all midterms, the different feedback types showed no statistically significant differences in performance, though there were some trends suggesting potential benefits worth further investigation. The system’s most substantial impact emerged through its required confidence ratings and explanations, which students reported transferring to their actual exam strategies. Approximately 40% of students engaged with textbook references when prompted by feedback—significantly higher than traditional reading compliance rates. Survey data revealed high student satisfaction (M=4.1/5), with 82.1% reporting increased confidence on midterm topics they had practiced, and 73.4% indicating they could recall and apply specific concepts from practice sessions. Our findings demonstrate how thoughtfully designed AI-enhanced systems can scale formative assessment while promoting sustainable study practices and self-regulated learning behaviors, suggesting that embedding structured reflection requirements may be more impactful than sophisticated feedback mechanisms. † † cc-license: by 1. Introduction It is a truism among faculty that students do not read the textbook ( Clump et al. 2004 ; French et al. 2015 ) . In contrast, our experience is that students are strongly motivated to tackle practice exams in the run-up to actual exams ( Carpenter 2012 ; Roediger and Butler 2011 ) . This paper describes an experimental deployment, in a class of 1002 students, of a system we developed that aimed to turn practice exams into a teaching tool. While the AI features meant to be the system’s centerpiece did not drive improvements in student exam performance, two other deliberate design choices aimed at shaping student engagement with the AI feedback—requiring students to 1) rate their confidence and 2) explain their reasoning while attempting every question—were unexpectedly impactful. These features had unanticipated positive effects on students’ metacognition, internal motivation, and long-term exam preparation strategies. To make multiple-choice practice exams more interactive we developed a tool that could give students formative feedback ( Hattie and Timperley 2007 ; Kulhavy et al. 1985 ) on their answer choices, rather than simple right and wrong responses. We used a Large Language Model (LLM) to either (i) guide students to an appropriate place in the textbook to acquire the knowledge they needed to answer correctly or (ii) generate textual guidance about the material the student was getting wrong. To make this feedback more targeted, our design required that each student explain their answer when they submitted it; this explanation could help the LLM understand what the student did not understand, in order to steer them to the right textbook section or to generate suitable corrective guidance. Based on analysis of prior educational feedback literature, we also surmised that it would be beneficial to record students’ confidence in their answer, and use that to further customize the feedback ( Kulhavy et al. 1985 ; Hattie and Timperley 2007 ) . We ran a randomized controlled trial in a 1002-student general biology class, comparing these two feedback approaches (guide-to-textbook and AI-generated feedback) to a control that only told students whether their answers were correct. Contrary to our expectations based on prior work on elaborated feedback, we found no significant difference between treatment conditions. Instead, the most substantial impact came from the metacognitive requirements we had initially included as supporting elements for the feedback system. This finding suggests that structured reflection may be more valuable for student learning than the specific content of feedback provided. While the benefits of practice testing are well-documented ( Carpenter 2012 ; Roediger and Butler 2011 ) , providing effective, personalized feedback to large student populations remains a significant challenge ( Butler et al. 2007 ; Wisniewski et al. 2020 ) . The emergence of sophisticated AI systems capable of generating contextual feedback offers promising new approaches to this challenge ( Tang and Zhan 2021 ; Ward et al. 2024 ) , especially when considered within the framework of self-regulated learning and student success ( Hattie and Timperley 2007 ; Wisniewski et al. 2020 ) , though careful integration is needed to balance AI assistance with student agency ( Ahmad and Ma 2024 ) . Traditional practice exam approaches face several key limitations. Static feedback and delayed grading often fail to provide students with the timely guidance they need ( Kulik and Kulik 1988 ; Butler et al. 2007 ) , while faculty time constraints make it difficult to offer detailed, personalized feedback to hundreds of students ( Smolansky et al. 2023 ) . Furthermore, students frequently focus on their numerical grades rather than developing deeper understanding of the course material ( Wisniewski et al. 2020 ; Vafeas 2013 ) . This tendency is exacerbated by the perceived disconnect between exam performance and specific textbook content, making it challenging for students to effectively address their knowledge gaps ( Klymkowsky and Cooper 2024 ; Chipperfield et al. 2022 ) . Because students are motivated by practice exams, we aimed to turn them into better teaching tools by providing customized feedback on student responses. The theory of effective feedback systems draws from several key education research areas. Self-regulated learning theory emphasizes the importance of helping students develop metacognitive strategies and self-assessment skills ( Hattie and Timperley 2007 ; Wisniewski et al. 2020 ) . Student confidence plays a crucial role in both learning and assessment, influencing how feedback is received and integrated ( Hattie and Timperley 2007 ; Kulhavy et al. 1985 ) . This work suggests that effective feedback systems should not only provide accurate information, but also support students in developing better study strategies and engaging more meaningfully with course materials. Despite growing interest in AI-enhanced educational tools, significant gaps remain in our understanding of their effectiveness. Limited research exists on AI-generated feedback in authentic learning contexts, particularly regarding how different types of feedback compare in supporting student learning. While research has established a connection between student confidence and the effectiveness of feedback ( Hattie and Timperley 2007 ; Kulhavy and Stock 1989 ) , its application in the context of AI-powered feedback remains

Read the complete text