Designing for Novice Debuggers: A Pilot Study on an AI-Assisted Debugging Tool Conference: 25th Koli Calling International Conference on Computing Education Research; November 11–16, 2025; Koli, Finland 25th Koli Calling International Conference on Computing Education Research (Koli Calling ’25), No

Oka Kurniawan, Erick Chandra, Christopher M. Poskitt, Yannic Noller, Kenny Tsu Wei Choo, Cyrille Jegourel · 2025

Read Designing for Novice Debuggers: A Pilot Study on an AI-Assisted Debugging Tool Conference: 25th Koli Calling International Conference on Computing Education Research; November 11–16, 2025; Koli, Finland 25th Koli Calling International Conference on Computing Education Research (Koli Calling ’25), No by Oka Kurniawan, Erick Chandra, Christopher M. Poskitt, Yannic Noller, Kenny Tsu Wei Choo, Cyrille Jegourel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21067.txt.

Introduction

Beginners learning to program often accept an AI's fix without understanding their bug. Oka Kurniawan and five co-authors design against that habit in a 2025 pilot study for the Koli Calling 2025 conference on computing education. Their tool is CodeHinter, an AI-assisted debugging assistant.

CodeHinter combines classic debugging tools with language-model techniques aimed at semantic errors. Rather than handing over a repair, it hints and prompts so students stay engaged in finding the fault. The paper reports findings from the second design iteration, tested with undergraduate students. The authors examine whether the tool helps novices fix code while still learning the debugging process itself. The study speaks to educators wary of AI shortcuts in programming courses.

Catalog record

CREATOROka Kurniawan, Erick Chandra, Christopher M. Poskitt, Yannic Noller, Kenny Tsu Wei Choo, Cyrille Jegourel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-21067

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Designing for Novice Debuggers: A Pilot Study on an AI-Assisted Debugging Tool Conference: 25th Koli Calling International Conference on Computing Education Research; November 11–16, 2025; Koli, Finland 25th Koli Calling International Conference on Computing Education Research (Koli Calling ’25), November 11–16, 2025, Koli, Finland DOI: 10.1145/3769994.3769997 ISBN: 979-8-4007-1599-0/25/11 CCS: Applied computing Education CCS: Software and its engineering Software testing and debugging CCS: Social and professional topics Software engineering education Oka Kurniawan, Erick Chandra, Christopher M. Poskitt, Yannic Noller, Kenny Tsu Wei Choo, Cyrille Jegourel Abstract . Debugging is a fundamental skill that novice programmers must develop. Numerous tools have been created to assist novice programmers in this process. Recently, large language models (LLMs) have been integrated with automated program repair techniques to generate fixes for students’ buggy code. However, many of these tools foster an over-reliance on AI and do not actively engage students in the debugging process. In this work, we aim to design an intuitive debugging assistant, CodeHinter , that combines traditional debugging tools with LLM-based techniques to help novice debuggers fix semantic errors while promoting active engagement in the debugging process. We present findings from our second design iteration, which we tested with a group of undergraduate students. Our results indicate that the students found the tool highly effective in resolving semantic errors and significantly easier to use than the first version. Consistent with our previous study, error localization was the most valuable feature. Finally, we conclude that any AI-assisted debugging approach should be personalized based on user profiles to optimize their interactions with the tool. † † cc-license: by 1. Introduction The first hurdle novice programmers encounter when writing code is ensuring that it is free of syntax errors ( Smith and Rixner 2019 ) . Although most compilers and interpreters generate error messages that provide information about these errors, many novice programmers struggle to interpret and fix them ( Denny et al. 2012 ) . To address these challenges, numerous tools and approaches have been developed to help novice programmers read, interpret, and resolve error messages effectively ( Campbell et al. 2014 ; Leinonen et al. 2023 ) . However, many of these tools primarily focus on improving error messages rather than guiding students through the debugging process ( Becker et al. 2019 ) . Recent advances in AI-assisted programming have introduced large language model (LLM) tools as potential debugging assistants, capable of automatically identifying and fixing errors. LLMs have already shown strong accuracy in correcting syntax errors, opening up new possibilities for automated debugging support ( Zhang et al. 2023b ; Joshi et al. 2023 ) . Moreover, LLM-based tools such as ChatGPT and GitHub Copilot can provide explanations of syntax errors along with code examples that help learners understand and correct their mistakes ( OpenAI 2022 ; GitHub 2023 ) . While LLM-based tools provide valuable assistance in fixing syntax errors, their ability to help novice programmers identify, understand, and resolve semantic errors remains limited. First, studies have shown that while LLM-based tools can resolve some semantic errors, their accuracy varies depending on the complexity of the problem ( Zhang et al. 2023b ) . Second, the generated solutions may differ significantly from students’ original buggy code, making it difficult for novice programmers to understand how to modify their code accordingly ( Zhang et al. 2024a ) . Third, LLM-based tools may also introduce new errors, which novice programmers often struggle to recognize or correct, leading to a negative learning experience. Finally, and most importantly, many LLM-based tools generate complete solutions to semantic errors rather than guiding users through the step-by-step debugging process that is essential for learning. As a result, novice programmers may develop an over-reliance on AI tools, which can hinder their ability to debug independently ( Collins 2023 ) . This dependence not only affects the accuracy of solutions but also impairs the ability of students to develop critical problem-solving skills. Without structured guidance, students may adopt AI-generated solutions without fully understanding how to troubleshoot and resolve issues on their own. Proficiency in testing, debugging, and fixing code is a fundamental skill for computer science (CS) graduates, and its importance is even greater in the era of AI-assisted programming, where AI tools enhance productivity but cannot substitute for a programmer’s ability to independently reason through errors. Our work introduces CodeHinter , an interactive debugging tool that goes beyond automatic error correction to actively guide students through diagnosing and fixing their own code. By integrating fault localization techniques with LLM-powered hints directly into the IDE through a simple one-click interface, CodeHinter promotes step-by-step problem solving, deepens students’ understanding of errors, and reduces reliance on fully automated solutions. We evaluate its usability through a user study, highlighting how structured, interactive guidance can strengthen debugging skills and improve programming education. 2. Related Work In this section, we review related work on how LLMs assist novice programmers in fixing errors. We examine various tools designed to help them resolve both syntax and semantic errors. Given the extensive literature on this topic, we focus on a selection of representative studies most relevant to our work. Leinonen et al. utilized Codex, a model based on GPT-3, to improve the clarity of programming error messages ( Leinonen et al. 2023 ) . Programming error messages are often difficult for novice programmers to interpret ( Prather et al. 2018 ) . Their work aimed to enhance these messages by providing explanations of the errors along with suggested fixes. They found that LLMs can generate helpful, novice-friendly explanations, improving students’ comprehension and their ability to correct errors. While the explanations generated by Codex were generally comprehensible, the suggested fixes were correct only 33% of the time. To improve the accuracy of generated fixes, Phung et al. introduced a novel runtime validation mechanism to assess whether the feedback provided by the LLM in the initial stage was suitable for students ( Phung et al. 2023 ) . Their approach involved iteratively querying the LLM to generate fixes for the buggy program while leveraging feedback from previous LLM iterations. If the number of syntactically correct programs exceeded a predefined threshold, the feedback was deemed acceptable. This method enabled them to achieve high precision in identifying and correcting syntax errors. However, their study was limited to syntax errors, and its effectiveness in addressing semantic errors has not been explored. A widely explored approach for addressing semantic errors is automated program repair (APR) ( Zhang et al. 2023a ) . LLMs have been applied to APR through fine-tuning, few-shot learning and zero-shot learning ( Zhang et al. 2024b ) . One notable example is PyDex, which employs few-shot learning to repair both syntactic and semantic bugs in introductory Python assignments ( Zhang et al. 2024a ) . PyDex utilizes the structure of a student’s buggy program as input to the LLM, resulting in repairs that require fewer edits. When compared to other APR tools, PyDex achieved a repair rate of 96.5%. Additionally, the average token edit distance for PyDex-generated patches was lower than that of competing tools. However, PyDex provides fixes without student intervention, which may hinder the development of debugging skills, as students receive corrections without actively engaging in the debugging process.

Read the complete text