AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study Conference: Proceedings of the 31st ACM Conference on Innovation and Technology in Computer Science Education V. 1; July 10–15, 2026; Madrid, Spain Proceedings of the 31st ACM Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2026), July 10–15, 2026, Madrid, Spain DOI: 10.1145/3803400.3809346 ISBN: 979-8-4007-2634-7/2026/07 CCS: Social and professional topics Computing education CCS: Applied computing Computer-assisted instruction
Yuri Noviello, Naaz Sibia, Anastasiia Birillo, Thomas Overklift Vaupel Klein, Michael Liut, Gosia Migut · 2026
Read AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study Conference: Proceedings of the 31st ACM Conference on Innovation and Technology in Computer Science Education V. 1; July 10–15, 2026; Madrid, Spain Proceedings of the 31st ACM Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2026), July 10–15, 2026, Madrid, Spain DOI: 10.1145/3803400.3809346 ISBN: 979-8-4007-2634-7/2026/07 CCS: Social and professional topics Computing education CCS: Applied computing Computer-assisted instruction by Yuri Noviello, Naaz Sibia, Anastasiia Birillo, Thomas Overklift Vaupel Klein, Michael Liut, Gosia Migut (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03288.txt.
Introduction
Noviello and co-authors test AI-made animated traces as a teaching aid for novice programmers. First programming courses often fail to make program execution clear. Visual aids can help, but proof for AI-made ones is thin. Their traces, called GATs, are narrated animations built on analogies. They align source code, run state, and a conceptual analogy in one view.
The study runs at two institutions, in Python and Java courses. Over 1,100 students took part in total. GATs are compared with plain text explanations. Measures cover quick learning, student experience, end-of-course engagement, and exam scores. GATs gave selective gains in quick learning. The gains were context-bound and short-lived. Learner engagement profiles shaped the effect a lot. Some students gained, others did not. The authors stress that such aids need personal fit. One format for all novices is not enough, even when the format is rich.
Catalog record
| CREATOR | Yuri Noviello, Naaz Sibia, Anastasiia Birillo, Thomas Overklift Vaupel Klein, Michael Liut, Gosia Migut |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03288
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study Conference: Proceedings of the 31st ACM Conference on Innovation and Technology in Computer Science Education V. 1; July 10–15, 2026; Madrid, Spain Proceedings of the 31st ACM Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2026), July 10–15, 2026, Madrid, Spain DOI: 10.1145/3803400.3809346 ISBN: 979-8-4007-2634-7/2026/07 CCS: Social and professional topics Computing education CCS: Applied computing Computer-assisted instruction Yuri Noviello, Naaz Sibia, Anastasiia Birillo, Thomas Overklift Vaupel Klein, Michael Liut, Gosia Migut Abstract . Introductory programming (CS1) courses often struggle to support students’ understanding of program execution. While visualizations can make execution processes explicit, their effectiveness depends on design and context, and empirical evidence for AI-generated visualizations remains limited. We propose Generated Animated Traces ( GATs ), AI-generated, analogy-based, narrated animations that coordinate source code, execution state, and conceptual analogies. We conduct a study at two institutions in CS1 courses (Python, N = 961 N=961 ; Java N = 151 N=151 ) comparing GATs to textual explanations. We measure immediate learning performance and experience, end-of-course engagement and exam performance. Results show that GATs can yield selective benefits for immediate learning, but benefits are context-dependent and short-term. We observe that GATs ’ influence on performance is moderated by learner engagement profiles. This finding underscores the importance of personalized approaches. † † cc-license: by Figure 1 . Study pipeline. Unique materials for the treatment group are marked in blue, and unique materials for the control group are marked in pink. Shared materials for both groups are marked in green. 1. Introduction Understanding how programs execute is a central challenge in introductory programming (CS1) courses. Novice programmers may learn to produce correct outputs while still lacking robust mental models of how variables, memory, and control flow interact at runtime, which limits transfer and long-term learning ( 39 ) . A common instructional response is to make program execution more explicit through visual explanations, such as traces, diagrams ( 10 ; 42 ) , animations that present multiple views of execution ( 40 ) , or domain-related analogies ( 38 ) . Prior work suggests that coordinating verbal and non-verbal channels can support comprehension ( 34 ) , while dynamic visualizations with attention guidance help learners focus on task-relevant elements ( 11 ) . Advances in Large Language Models (LLMs) now make it possible to automatically generate such instructional materials at scale, creating new opportunities to deploy rich visual explanations consistently across large-enrollment courses ( 32 ; 18 ) . However, the educational impact of such materials remains unclear. Empirical evidence is limited on when, for whom, and under what conditions AI-generated instructional materials support learning, particularly across institutional contexts and learner profiles ( 18 ) . Prior work further suggests that instructional interventions rarely benefit all learners equally, and that average effects may obscure meaningful differences in engagement and learning outcomes ( 17 ) . In this work, we investigate Generated Animated Traces ( GATs ), an AI-generated, analogy-based, narrated animation that synchronizes a code snippet, a memory diagram, and a conceptual analogy within a single presentation to explicitly represent how program state and control flow evolve during execution. Rather than assuming benefits from coordinated views, we empirically evaluate how GATs compare to carefully matched step-by-step textual explanations across two CS1 courses with different languages and course structures. We address three research questions: RQ1: How do GATs affect (a) immediate post-intervention programming task performance and (b) course summative performance, compared to textual explanations? RQ2: How does students’ learning experience differ between GATs and textual explanations, as measured by self-reported frustration, cognitive load, and situational interest? RQ3: To what extent are the effects of GATs moderated by learner engagement profiles? In this work we provide the first systematic evaluation of GATs for CS1 across: (1) two institutions with different languages and structures, (2) multiple instruments (immediate/summative performance, learning experience, engagement), and (3) learner heterogeneity via engagement profiles. Our findings show that the impact of GATs is context- and learner-dependent, underscoring the need to move beyond average effects in CS1 instruction. 2. Related Work Learning to program requires more than producing correct outputs. It requires building mental models that link code to runtime behavior, including variable state, control flow, and memory operations ( 22 ; 40 ) . In CS1, instruction is commonly scaffolded through notional machines, idealized models of how programming languages execute, that help learners reason about scope, state changes, and execution order ( 13 ) . Effective learning, therefore, depends on supporting abstraction rather than surface-level code tracing alone. Program visualizations are widely used instructional tools for making program execution explicit. Tools such as Python Tutor expose step-by-step execution traces and runtime state ( 16 ) , which can support tracing accuracy. However, highly detailed visualizations may anchor novices to low-level mechanics, limiting abstraction and transfer ( 40 ; 12 ) . Learners may successfully follow traces without developing deeper conceptual understanding. However, learners do not automatically integrate multiple representations when exposed to program visualizations. In an eye-tracking study, novices often attend selectively to a single view unless explicitly guided ( 2 ) . Synthesizing this literature, Sorva concludes that visualization effectiveness depends not on visual richness alone, but on how representations are coordinated ( 40 ) . Similarly, to avoid redundancy and cognitive overload, the DeFT framework emphasizes that multiple external representations require careful design ( 1 ) and should guide attention to task-relevant elements ( 41 ; 30 ) . Also, the expertise-reversal effects suggest that instructional supports that are helpful for novices may hinder more advanced learners ( 24 ) . These findings underscore that adding instructional detail does not inherently improve learning outcomes. In addition to program visualizations, analogies are frequently used in CS1 to ground abstract execution concepts in familiar domains. Well-designed analogies have been shown to reduce cognitive load and improve conceptual understanding ( 14 ; 7 ) . However, their effectiveness is highly design-sensitive, and misaligned analogies can mislead learners or impede abstraction ( 5 ) . At the same time, the analogy-based videos may increase engagement even when performance gains are limited ( 45 ) , highlighting a distinction between learning experience and measured outcomes. As an extension of visualization- and analogy-based instruction, recent studies indicate that AI-generated teaching materials, when pedagogically sound, can be as effective as human-produced materials ( 31 ; 44 ) , although they tend to score lower on social presence ( 18 ) . Also, AI can assist learners in generating personalized analogies ( 4 ; 19 ) . As AI lowers the cost of producing rich instructional media for various needs, there is growing interest in deploying such materials at scale ( 18 ) . However, empirical evidence remains limited on when, for whom, and under what conditions AI-generated instructional materials support learning, particularly across institutional contexts and learner profiles. Addressing this gap requires evaluations that move beyond average effects to consider learner heterogeneity and multiple learning outcomes. 3. Methods 3.1. Study design Overview. In 2025, we conducted two similar studies at Delft University of Technology (TU Delft) and the University of Toronto Mississauga (UofT), both research-intensive publicly funded institutions. Students