Starting Seatwork Earlier as a Valid Measure of Student Engagement

Ashish Gurung Jionghao Lin Zhongtian Huang, Conrad Borchers Ryan S. Baker Vincent Aleven, Kenneth R. Koedinger · 2025

Read Starting Seatwork Earlier as a Valid Measure of Student Engagement by Ashish Gurung Jionghao Lin Zhongtian Huang, Conrad Borchers Ryan S. Baker Vincent Aleven, Kenneth R. Koedinger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13341.txt.

Introduction

Ashish Gurung and six colleagues propose using when students start seatwork as a measure of engagement. Automated detectors built from learning software logs already track self-regulation, but start and stop decisions are underexplored. The team computed session-level measures such as delayed start and early stop from year-long Cognitive Tutor logs for 222 students in grades eight through twelve.

These measures proved reliable month to month and improved prediction of final math scores beyond prior knowledge and gaming-the-system behavior. A second analysis in the i-Ready system, with 818 seventh graders, showed the measures also predict state test scores. The authors argue such log-based measures capture motivation without extra surveys or class disruption.

Catalog record

CREATORAshish Gurung Jionghao Lin Zhongtian Huang, Conrad Borchers Ryan S. Baker Vincent Aleven, Kenneth R. Koedinger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13341

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Starting Seatwork Earlier as a Valid Measure of Student Engagement Ashish Gurung Jionghao Lin Zhongtian Huang, Conrad Borchers Ryan S. Baker Vincent Aleven, Kenneth R. Koedinger Abstract Prior work has developed a range of automated measures (“detectors”) of student self-regulation and engagement from student log data. These measures have been successfully used to make discoveries about student learning. Here, we extend this line of research to an underexplored aspect of self-regulation: students’ decisions about when to start and stop working on learning software during classwork. In the first of two analyses, we build on prior work on session-level measures (e.g., delayed start, early stop) to evaluate their reliability and predictive validity. We compute these measures from year-long log data from Cognitive Tutor for students in grades 8–12 (N = 222). Our findings show that these measures exhibit moderate to high month-to-month reliability (G > .75), comparable to or exceeding gaming-the-system behavior. Additionally, they enhance the prediction of final math scores beyond prior knowledge and gaming-the-system behaviors. The improvement in learning outcome predictions beyond time-on-task suggests they capture a broader motivational state tied to overall learning. The second analysis demonstrates the cross-system generalizability of these measures in i-Ready, where they predict state test scores for grade 7 students (N = 818). By leveraging log data, we introduce system-general naturally embedded measures that complement motivational surveys without extra instrumentation or disruption of instruction time. Our findings demonstrate the potential of session-level logs to mine valid and generalizable measures with broad applications in the predictive modeling of learning outcomes and analysis of learner self-regulation. keywords Self-Regulation, Motivation, Discovery with Models, K-12 1 Introduction Self-regulated learning (SRL) [ 60 ] has been a cornerstone in the development of automated detectors of student engagement in Educational Data Mining (EDM). Given the nature of logged data from Digital Learning Platforms (DLP), EDM researchers have frequently leveraged time-based measures to capture key aspects of self-regulation, such as motivation, time management, and goal orientation. These time-based measures have been used to contextualize various aspects of learner behaviors in the development of detectors for effort [ 67 , 38 ] , attention [ 79 , 59 ] , gaming-the-system [ 55 ] , mind-wandering [ 18 ] , and off-task behavior [ 68 , 5 ] . Follow-up research has demonstrated the predictive utility of these measures, linking them to students’ immediate [ 20 , 68 , 38 ] , short-term, and long-term learning outcomes [ 56 , 65 ] . While EDM and Learning Analytics have extensively examined process-level engagement—focusing on interactions with tasks or materials—session-level exploration has been generally limited to students working independently in higher education [ 35 , 31 , 48 , 45 ] . Despite an inherent expectation for student participation in K-12 settings, session-level engagement remains largely underexplored. Beyond a few studies on session-level time-based measures as indicators of diligent learner behavior [ 23 , 25 ] , its broader implications are not well understood. As such, dedicated classwork sessions provide a valuable opportunity to investigate session-level behaviors in structured learning environments. Student behavior during classwork sessions can be influenced by individual self-regulation [ 81 , 36 ] as well as broader classroom dynamics, and teacher support [ 33 , 60 ] . For instance, individual goal orientation [ 36 , 50 ] as well as teacher instructions, peer engagement, or other systemic factors [ 49 , 17 , 60 ] can influence a student’s decision about when to start. Addressing this gap, especially for classwork sessions, is essential for understanding how classroom dynamics shape individual learning behaviors and overall learning outcomes. Additionally, combining session-level and process-level engagement can provide deeper insights into how behaviors at both levels influence broader engagement and learning. In this light, we build on prior research leveraging session-level time-based measures [ 23 , 77 ] to examine their reliability and predictive validity in relation to learning outcomes, aiming to identify student differences in diligent time use. Specifically, we analyze how students’ behavior, such as delayed starting, session length, and early stopping when using DLP during classwork, manifests across students and influences their learning outcomes, including year-end grades and performance on standardized state tests. Through these session-level measures, we contribute to the growing body of EDM and Learning Analytics research mining SRL differences from log data, offering a novel lens on student-initiated practice behavior in the context of a classwork session. 2 Background 2.1 Self-Regulated Learning The conceptualization of SRL as a theoretical construct has evolved over time, with several empirical studies contributing to its refinement [ 54 ] . Among the formalizations of SRL, the models proposed by Zimmerman [ 81 ] , Pintrich [ 60 ] , and Winne and Hadwin [ 74 ] are often referenced in EDM and Learning Analytics. These frameworks offer distinct yet complementary perspectives on factors and mechanisms influencing learner behavior. Zimmerman’s model emphasizes individual self-regulation, whereas Pintrich’s framework, as well as Winne and Hadwin’s framework, emphasize the importance of contextual influences (e.g., peer interactions, teacher guidance, and class norms), making it more suitable for our focus on K-12 classwork session data. Previous research has explored the influence of various SRL factors on self-regulatory processes. Goal setting, task value, and goal orientation influence engagement, with mastery goals fostering deep learning [ 28 ] and performance goals prioritizing learning outcomes [ 50 , 36 ] . Motivation and self-efficacy underpin all phases, shaping persistence and self-regulation [ 75 , 30 ] , while high self-efficacy fosters resilience, helping learners reframe challenges as opportunities [ 73 ] . Additionally, time management, effort regulation, and metacognitive monitoring sustain engagement by guiding progress tracking, strategic adjustments, and reflection [ 72 , 4 ] . In practice, effective self-regulation is often observed through learners’ adjustments in how they manage time and effort. For example, Gollwitzer et al. [ 36 ] , in their meta-analysis, highlighted how goal-oriented, explicit “if-then” plans shape student self-regulation. For instance, a student might plan, “If my friends distract me, then I will ignore them and keep working.” These situational cues helped students effectively regulate their learning by developing strategic goal-directed plans for seizing opportunities( d = 0.61) and shielding them from unwanted influences ( d = 0.77). 2.2 Time-Based Detectors Different dimensions of SRL have directly or indirectly informed the development of time-based measures (i.e., detectors) to capture various aspects of learner behavior in DLPs. These measures provide valuable insight into various aspects of student engagement, such as effort [ 67 , 38 ] , off-task behavior [ 68 , 5 ] , and procrastination [ 12 , 78 , 64 ] . Learner time management also plays a crucial role in the modeling and detection of more complex behaviors, such as gaming-the-system [ 7 , 55 ] , wheel-spinning (i.e., unproductive persistence) [ 8 , 37 ] , attention [ 79 , 59 ] , and mind-wandering [ 18 , 51 ] . For instance, the rule-based gaming-the-system detector by Paquette et al. [ 55 ] uses 19 contextually interpretable actions, 13 of which ( \sim 68%) directly incorporate time to establish context. For example, a help request within five seconds of starting a problem would indicate: “the student did not

Read the complete text