Assessing the Impact and Underlying Pathways of Sequenced AI feedback on Student Learning

Jie Cao, Chloe Qianhui Zhao, Christian Schunn, Elizabeth A. McLaughlin, Jionghao Lin, Kenneth R. Koedinger · 2026

Read Assessing the Impact and Underlying Pathways of Sequenced AI feedback on Student Learning by Jie Cao, Chloe Qianhui Zhao, Christian Schunn, Elizabeth A. McLaughlin, Jionghao Lin, Kenneth R. Koedinger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07469.txt.

Introduction

Cao and five colleagues test sequenced AI feedback, which gives hints before answers, against direct feedback in 199 students. Sequenced feedback raised engagement a bit and felt more encouraging and autonomy-supporting, but it also raised mental effort. A good mood pathway from encouragement was fully cancelled by a bad behavior pathway tied to tasks needing three or more tries. Net learning was significantly poorer with sequenced feedback, a caution for feedback design.

Catalog record

CREATORJie Cao, Chloe Qianhui Zhao, Christian Schunn, Elizabeth A. McLaughlin, Jionghao Lin, Kenneth R. Koedinger
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55666
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.07469

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Assessing the Impact and Underlying Pathways of Sequenced AI feedback on Student Learning Jie Cao, Chloe Qianhui Zhao, Christian Schunn, Elizabeth A. McLaughlin, Jionghao Lin, Kenneth R. Koedinger Abstract Feedback is essential for learning, but its effectiveness relies heavily on how well it engages students in the educational process. Generative AI offers novel opportunities to efficiently produce rich, formative feedback, ranging from direct explanations to incrementally sequenced scaffolding designed to promote learner autonomy. Despite these capabilities, it is still unclear whether sequenced (layered) AI feedback—which provides encouragement and hints before revealing the correct answer—genuinely enhances engagement and learning outcomes. To investigate this, we randomly assigned 199 participants to receive either sequenced or non-sequenced AI-generated feedback. We evaluated its impact on learning performance, cognitive and behavioral engagement, and affective perceptions to understand how these factors mediate overall learning outcomes. Results show that sequenced feedback elicited slightly higher behavioral engagement and, as anticipated, was perceived as more encouraging and supportive of student independence. Concurrently, however, it induced a higher level of mental effort. Mediation analyses identified a positive affective pathway driven by perceived encouragement, which was completely counteracted by a negative behavioral pathway associated with the average number of tasks requiring three or more submissions; the cognitive pathway (mental effort) remained non-significant. Overall, sequenced feedback led to significantly poorer learning outcomes when compared to direct, non-sequenced feedback. These findings highlight a crucial trade-off: although sequenced AI scaffolding boosts engagement and positive user perceptions, it can have a detrimental effect on actual learning performance. By integrating analyses of outcomes, perceptions, and underlying mechanisms, this study provides nuanced insights for designing automated, AI-driven feedback systems. † † highlights: sequenced AI feedback led to poorer learning performance than non-sequenced AI feedback It boosted perceived encouragement and independence, but raised mental effort. It also enhances behavioral engagement slightly. Encouragement aided learning, while frequent submission behaviors hindered it. Design must optimize positive tone, limit gaming, and aid cognitive processing. 1 Introduction Feedback is widely recognized as critical for supporting knowledge acquisition, sustaining motivation, and promoting self-regulation ( 24 ) . As found in a recent meta-analysis, feedback exerts a moderate positive effect on learning outcomes across different educational levels and domains ( 58 ) . Yet, not all feedback is equally effective. Its impact depends on contextual factors such as the learning environment and learner characteristics ( 25 ) , as well as intrinsic features of the feedback itself, including modality, delivery mode, source, and, most importantly, its content ( 38 ; 22 ; 41 ) . A particularly effective form of feedback provides the correct answer with an explanation of why it is the correct answer ( 25 ; 55 ; 10 ; 58 ) . Such elaborated feedback with the correct answer can deliver information directly and efficiently, and it can be further elaborated to include explanations, strategies, and guidance for self-regulation ( 58 ) , which shows effectiveness in producing larger learning gains, particularly for students with lower prior knowledge ( 25 ) . Nevertheless, important limitations remain. During the learning process, immediate provision of correct answers may encourage students to copy the answer without fully engaging in understanding ( 31 ) . Emerging learner-centered feedback theories emphasize that feedback should not merely transmit information, but rather foster active engagement, self-regulation, and shared responsibility in the feedback process ( 42 ) . Effective feedback, from this perspective, with the feature of learner-centered, requires them to generate, process, and respond actively to information, while also receiving affective support such as encouragement ( 13 ) . These insights suggest that withholding the correct answer initially, while offering encouragement and prompts for self-regulation, may promote deeper engagement. Building on this reasoning, the present study introduces a dynamic layered feedback design (refers to the sequenced feedback) that integrates the strengths of learner-centered feedback and elaborated feedback with correct answers (See Figure 1 and details in 3.2.2 ). In this design, the first layer provides encouragement, promotes agency and independence, and includes actionable suggestions without revealing the correct answer, while a subsequent layer supplies elaborated, correct answer-based feedback if learners continue to struggle. This approach to layering information is related to the sequential hint approach included in many Intelligent Tutoring Systems (ITS) systems that provide layer hints for students from basic pointing hints to final “bottom-out hints” (i.e., hints that provide answers, typically found at the end of a hint sequence) ( 56 ) . However, different from feedback, these hints in ITS are often provided at the student’s request ( 30 ; 2 ) rather than depending up the submission of a solution for feedback. This distinction is important because some students ask for hints before any struggle or tend to request bottom-out hints while paying minimal attention to preceding hints, resulting in less learning ( 32 ; 1 ) . Layered feedback, therefore, has the potential to promote greater independence and deeper engagement during learning ( 56 ) . Figure 1: The design of the layered feedback compared with the non-layered feedback At the same time, with the rapid development of online higher education and the surge in online learners, there has been a substantial increase in the demand for feedback. However, providing immediate feedback at scale is often infeasible for educators. Advances in large language models (LLMs) offer unprecedented opportunities to automate feedback generation that is rapid, adaptive, and multimodal ( 12 ; 27 ; 54 ) . Further, the feedback can be easily elaborated to include personalized content. However, too much elaboration can be overwhelming. Generally, concerns have been voiced regarding AI providing direct answer information that guides task completion without requiring any learning ( 29 ) . Prior studies have demonstrated both the feasibility and quality of LLM-generated feedback. However, existing work has typically relied on uniform prompts that produce structurally similar feedback throughout learning ( 33 ; 50 ) . On the one hand, there is robust evidence that elaborate feedback with correct answers facilitates learning ( 25 ; 55 ; 58 ) . On the other hand, learner-centered feedback theory ( 42 ) and studies on layered hints in ITS suggest that layered feedback holds significant potential for fostering independent and deep engagement ( 56 ) . Consequently, it remains to be determined whether layered LLM-generated feedback that combines learner-centered elements with delayed correct answers can yield superior learning outcomes compared to elaborated LLM-generated feedback. Furthermore, previous studies on LLM-generated feedback have primarily compared academic performance and user perceptions against human feedback (e.g., ( 53 ; 17 ) ). Consequently, there remains a lack of comparative analysis between different LLM-generated feedback, and the underlying mechanisms(how different feedback versions influence learning engagement, then finally influence final outcomes) remain unexplored. Thus, to address these gaps, we compared the effect of LLM-generated layered feedback and non-layered feedback on academic performance in an online higher education context, and investigated how learners’ behavioral, cognitive engagement as well as affective perception differ in two feedback conditions and how do these differences lead to academic performance differences. Specifically, we propose four Research Questions (RQs): 1. RQ1: Does layered feedback significantly improve learners’ learning performance? 2. RQ2: How do learners behaviorally engage with layered feedback compared to non-layered feedback? 3.

Read the complete text