Evolution of Programmers’ Trust in Generative AI Programming Assistants Conference: Proceedings; Date; Koli, Finland 25th Koli Calling International Conference on Computing Education Research DOI: XXXX ISBN: XXX CCS: Social and professional topics Software engineering education CCS: Human-centered c

Anshul Shah, Thomas Rexin, Elena Tomson, Leo Porter, William G. Griswold, Adalbert Gerald Soosai Raj · 2025

Read Evolution of Programmers’ Trust in Generative AI Programming Assistants Conference: Proceedings; Date; Koli, Finland 25th Koli Calling International Conference on Computing Education Research DOI: XXXX ISBN: XXX CCS: Social and professional topics Software engineering education CCS: Human-centered c by Anshul Shah, Thomas Rexin, Elena Tomson, Leo Porter, William G. Griswold, Adalbert Gerald Soosai Raj (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13253.txt.

Introduction

Anshul Shah, Thomas Rexin and four co-authors track in 2025 how student trust in Copilot shifts with use. Over-trust risks flawed code; under-trust wastes gains. Trust moves with time, so snapshots mislead.

Seventy-one senior students used Copilot on legacy code for one hour, then ten days. Surveys plus interviews gauge levels and reasons. Mean trust rose over time. After the project, students still saw a need for skilled manual work on some tasks. Seeing correct output, grasping context use, and learning language basics lifted trust. Four teaching moves close the study.

Catalog record

CREATORAnshul Shah, Thomas Rexin, Elena Tomson, Leo Porter, William G. Griswold, Adalbert Gerald Soosai Raj
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-13253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evolution of Programmers’ Trust in Generative AI Programming Assistants Conference: Proceedings; Date; Koli, Finland 25th Koli Calling International Conference on Computing Education Research DOI: XXXX ISBN: XXX CCS: Social and professional topics Software engineering education CCS: Human-centered computing Empirical studies in HCI CCS: Social and professional topics Computing literacy Anshul Shah, Thomas Rexin, Elena Tomson, Leo Porter, William G. Griswold, Adalbert Gerald Soosai Raj Abstract . Motivation. Trust in generative AI programming assistants is a vital attitude that impacts how programmers use those programming assistants. Programmers that are over-trusting may be too reliant on their tools, leading to incorrect or vulnerable code; programmers that are under-trusting may avoid using tools that can improve their productivity and well-being. Methods. Since trust is a dynamic attitude that may change over time, this study aims to understand programmers’ evolution of trust after immediate (one hour) and extended (10 days) use of GitHub Copilot. We collected survey data from 71 upper-division computer science students working on a legacy code base, representing a population that is about to enter the workforce. Leveraging existing survey instruments and open-ended free response questions, we quantitatively measure student trust levels and qualitatively uncover why student trust changes. Findings. Student trust, on average, increased throughout the study. After completing a project with Copilot, however, students felt that Copilot requires a competent programmer to complete some tasks manually. Students mentioned that seeing Copilot’s correctness, understanding how Copilot uses context from the code base, and learning some basics of natural language processing contributed to their elevated trust. On the other hand, students pointed to Copilot’s incorrect code suggestions as one of their main reasons for having less trust in the tool. Implications. Our study helps instructors and industry managers understand the factors that influence how students calibrate their trust with generative AI programming assistants. We make four pedagogical recommendations, which are that CS educators should 1) provide opportunities for students to work with Copilot on challenging software engineering tasks to calibrate their trust, 2) teach traditional skills of comprehending, debugging, and testing so students can verify output, 3) teach students about the basics of natural language processing, and 4) explicitly introduce and demonstrate the range of features available in Copilot. 1. Introduction Adoption of generative AI assistants is growing among the software engineering industry, with GitHub Copilot in particular being the most widely-adopted AI tool among businesses ( Dohmke 2023 ) . Whether our students join start-up companies or established tech companies, it is highly likely that they will use generative AI programming assistants in their work ( Mehta 2025 ; Langley 2024 ) . Trust in these AI programming assistants is a vital attitude for educators and researchers to consider, since a programmers’ trust in an AI tool impacts their reliance on the tool ( Liao and Sundar 2022 ; Wang et al. 2024 ) . Developers who are over-trusting of their AI assistants may introduce insecure or incorrect code ( Perry et al. 2023 ; Prather et al. 2023 ; Amoozadeh et al. 2024 ) , while developers who are under-trusting may miss out on benefits of AI assistants, such as improved productivity and developer well-being ( Shihab et al. 2025 ; Cui et al. 2024 ; Peng et al. 2023 ; Shani 2023 ) . Our study aims to understand how and why student trust changes as they learn about and use GitHub Copilot. While the majority of research on student attitudes and use of generative AI assistants has focused on introductory computer science courses (CS) ( Prather et al. 2023 ; Vadaparty et al. 2024 ; Denny et al. 2024 ; Prather et al. 2024 ; Amoozadeh et al. 2024 ) , we focus on upper-division CS students’ trust in AI assistants while working on an existing code base. Our findings aim to inform computing educators how to foster students’ AI literacy and calibrate students’ trust levels to prepare students for a software engineering field that is embracing generative AI assistants. Ozkaya 2023 summarizes the importance of adapting our instruction to align with the reality of the software engineering industry: “Software engineering and computer science education has already missed the boat by continuing to focus on teaching green field development while today the reality of system development is brownfield… We teach students hello world development, while we should be teaching them how to read millions of lines of code, triage and fix bugs that they have not contributed to… With LLMs and their sister AI-driven apps assisting developers, we need to be teaching next-generation software engineers when to trust, how to create evidence to trust, how to do trust assessment rapidly and correctly, and how to improve such assistants.” ( Ozkaya 2023 ) To lay the groundwork for computing educators to teach students how to calibrate their trust with AI assistants while working on large code bases (i.e., brownfield development), we ask the following two research questions: RQ1:: How does student trust in GitHub Copilot change after immediate and extended use of GitHub Copilot in a large code base? RQ2:: What factors contribute to students’ evolution of trust in GitHub Copilot in a large code base? We studied how 71 upper-division CS students’ trust levels change 1) after attending a lecture in which they learned basics about natural language processing, saw demonstrations of features that are useful for brownfield programming, and worked with Copilot for a small task and 2) after completing a 10-day project where they added a feature to a large code base using GitHub Copilot. Using a survey to collect repeated measures of student trust and open-ended questions to understand why students’ trust changes, we found that, on average, trust in GitHub Copilot increased after immediate and extended use. Students’ immediate trust attitudes were influenced by Copilot’s code generation abilities, Copilot’s context-awareness features that support program comprehension, and understanding Copilot’s inner-workings. After extended Copilot use, however, students overwhelmingly highlighted the importance of having a “competent programmer” to verify or correct Copilot’s output. Our findings lead to four concrete recommendations for how CS instructors can teach students how to effectively and responsibly use AI assistants. 2. Theoretical Framework: The MATCH Model Figure 1. The MATCH model, reproduced exactly as presented by Liao and Sundar 2022 ( Liao and Sundar 2022 ) . Early models of trust formation emerged from social science research. The Ability-Benevolence-Integrity (ABI) model, developed in 1995 to explain interpersonal trust formation ( Mayer et al. 1995 ) , posits that the three characteristics of a trustee that impact trust are: 1) the trustee’s ability (i.e., competency) 2) the trustee’ benevolence (i.e., aligned intentions and goals between), and 3) the trustee’s integrity (i.e., adhering to a clear set of principles) ( Mayer et al. 1995 ) . In a similar vein, Lee and Moray 1992 ’s model of trust formation between humans and automation says that trust depends on a system’s performance (i.e., ability), purpose (i.e., benevolence), and process (i.e., integrity) ( Lee and Moray 1992 ) . One limitation of these models, however, is that they only describe characteristics of the trustee, or automated system. As a result, Sundar et al. 2008 presented the MAIN model in 2008 that highlighted the importance of

Read the complete text