To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks
Jessica Hutchison, Ian Tyler Applebaum, Kenneth Angelikas, Kush Rakesh Patel, Phuoc Nguyen, Antonio Lazaro, Nicholas Rucinski, Rahad Arman Nabid, Stephen MacNeil · 2026
Read To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks by Jessica Hutchison, Ian Tyler Applebaum, Kenneth Angelikas, Kush Rakesh Patel, Phuoc Nguyen, Antonio Lazaro, Nicholas Rucinski, Rahad Arman Nabid, Stephen MacNeil (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30549.txt.
Introduction
Jessica Hutchison and eight colleagues measure whether students actually think about the code an AI writes for them. AI completion tools suggest code that students accept with one keystroke. Accepting blindly short-circuits the reflection learning needs. Their tool, Clover, logs every interaction with suggestions. It also inserts attention checks during programming tasks. Those checks probe whether students are thinking as they work. A taxonomy of behavior metrics organizes the measurements.
Interaction patterns were compared against attention check performance. Task performance entered the analysis as well. Students who accepted suggestions more often scored worse on checks. Higher tab-accept rates tracked lower engagement. Longer dwell time before deciding pointed the other way. It was associated with higher attention check scores. The authors discuss using such process data in teaching. Attention checks might support reflection during AI-assisted programming. The findings give instructors measurable signals instead of impressions.
Catalog record
| CREATOR | Jessica Hutchison, Ian Tyler Applebaum, Kenneth Angelikas, Kush Rakesh Patel, Phuoc Nguyen, Antonio Lazaro, Nicholas Rucinski, Rahad Arman Nabid, Stephen MacNeil |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30549
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks Jessica Hutchison, Ian Tyler Applebaum, Kenneth Angelikas, Kush Rakesh Patel, Phuoc Nguyen, Antonio Lazaro, Nicholas Rucinski, Rahad Arman Nabid, Stephen MacNeil Abstract . AI code completion tools, such as Github Copilot, provide students with code suggestions to help them write programs. However, recent qualitative studies suggest that students fail to critically evaluate these suggestions. We present Clover, a code completion tool that logs students’ interactions with code suggestions and additionally offers attention checks to probe reflective engagement during programming tasks. We also develop a taxonomy of behavioral interaction metrics for AI-assisted programming, informed by literature. We analyzed relationships between interaction patterns, engagement with attention checks, and task performance. We observed that higher rates of tab accept were associated with lower attention check performance, while increased dwell time was associated with higher attention check performance. We conclude by discussing how programming process data and attention checks might support reflective engagement in AI-assisted programming. \author@bx@sep =0.5pc 1. Introduction AI-assisted programming is rapidly changing how software is written, with developers increasingly working alongside systems that generate code suggestions in real time. AI code completion tools, such as GitHub Copilot and Cursor, are increasingly integrated into professional workflows and, more recently, into computing courses as well ( 33 ; 24 ) . Although student adoption of code assistants has lagged behind general purpose AI tools such as ChatGPT, recent surveys show their use by computing students is steadily increasing ( 10 ; 15 ; 16 ; 25 ; 32 ) . As these tools become standard, fundamental questions arise about how students interact with AI code suggestions and how to meaningfully measure these interactions. Despite growing adoption, understanding students’ interactions with AI code completion tools remains a challenge. Existing studies rely on qualitative methods including think-aloud protocols ( 1 ; 35 ) , screen recordings ( 27 ; 31 ) , screenshots ( 21 ) , and process journals ( 30 ) . While these approaches provide rich insights they are difficult to scale and offer limited visibility into fine-grained real-time interactions with code suggestions. This measurement gap is important given emerging evidence that AI code assistants can impact learning behaviors and outcomes. Prior work has suggested that code completion tools can mislead students and disrupt their metacognitive processes ( 27 ; 23 ) . They can also lead to unproductive debugging ‘rabbit holes’ ( 34 ) . These negative impacts tend to disproportionately affect students with less experience and lower self efficacy ( 2 ; 26 ; 9 ; 36 ; 20 ) . Therefore, it is critical to develop metrics and methods for evaluating how students interact with these tools. To address this gap, we introduce Clover, a code completion tool that instruments student interactions with AI-generated suggestions and introduces lightweight attention checks to probe reflective engagement during programming tasks. To inform the metrics Clover tracks, we developed a comprehensive Taxonomy of Interaction Behaviors with AI Code Suggestions guided by metrics used across prior literature ( 28 ; 27 ; 1 ; 34 ; 14 ) . We then conducted a lab study with Clover in a CS1 course and analyze relationships between interaction patterns, attention check performance, and task performance to investigate the following research questions: RQ1: How does CS1 students’ usage of a code completion tool relate to their task performance? RQ2: How does CS1 students’ engagement with attention checks relate to their task performance? In this paper, we make the following contributions: • Clover, an instrumented code completion tool that logs fine-grained interactions with AI-generated suggestions and integrates attention checks to probe reflective engagement. • A taxonomy of interactions with AI code suggestions for AI-assisted programming synthesized from prior work. • An empirical study demonstrating how interaction patterns relate to engagement and task performance. 2. Related Work Programming Process Data (PPD) captures how students interact with the integrated development environment while coding. PPD can be collected at varying levels of granularity, from coarse-grained artifacts, such as final submissions, to fine-grained logs of individual keystrokes ( 5 ; 17 ) . PPD has been used to study how novice programmers code, identify sources of struggle, and support early intervention or integrity monitoring ( 29 ; 5 ; 17 ; 7 ; 3 ; 13 ) . However, this prior work has been focused on traditional programming environments without AI assistance. The introduction of AI code completion tools have surfaced diverse interaction patterns and complex impacts on learning. Prior work suggests that programmers interact with AI tools differently depending on their goals and experience. For example, 1 conducted one of the earliest studies of how professional programmers interact with Github Copilot identifying two modes: exploration where programmers use Copilot to explore options and acceleration where the programmer knows what to do and uses Copilot to get there faster. While this shows the flexibility of code assistants for experts, subsequent work suggests that novices may struggle to use these tools effectively. Based on an eye tracking study of students using Github Copilot, 27 found that less prepared students were often misled by AI suggestions and struggled to differentiate good suggestions from bad ones. This reflected a broader trend of unequal learning benefits associated with AI use ( 9 ; 25 ; 2 ; 27 ; 36 ) . Another study by 34 showed how students could get stuck in ‘debugging rabbit holes’ when using code generated by AI code assistants. Additional issues include students experiencing an ‘illusion of competence’ ( 27 ) where students have difficulty identifying gaps in understanding ( 27 ; 20 ) . However, understanding these interaction patterns remains challenging. Most studies have used qualitative methods such as screen capture ( 31 ) , eye-tracking ( 27 ) , or manual observations ( 31 ) . While these approaches provide valuable insights, they are labor-intensive and difficult to scale. PPD offers a complementary, scalable approach for analyzing behaviors such as accepting or rejecting suggestions across larger populations. However, existing PPD approaches have not been fully adapted to capture these interaction patterns. 3. Methods 3.1. Clover Design and Implementation Clover is an integrated development environment (IDE) extension that uses the Gemini 3 Large Language Model to provide real-time, single-line code suggestions at the cursor position based on code already in the editor. While programming, students can accept these code suggestions or write their own code manually. To capture how students engage with suggestions, Clover logs students’ interactions, which are described in detail in Section 3.3 . 3.1.1. Instrumenting the Copilot Experience To maximize ecological validity, Clover was implemented as a Visual Studio Code extension, mirroring the interface and functionality of widely-adopted tools like GitHub Copilot. We kept familiar interaction models, such as Tab-to-accept , with code suggestions appearing automatically when a user pauses during typing. By maintaining this user experience, students likely behave as they would in a real-world programming environment, allowing us to collect authentic behavioral data. 3.1.2. Single-Line Code Suggestions Large blocks of AI code suggestions can be difficult to understand and debug ( 34 ) , so Clover provides single-line code suggestions to reduce cognitive load and