Changes in Coding Behavior and Performance Since the Introduction of LLMs DOI: 10.1145/3785022.3785075 Conference: The 15th International Learning Analytics and Knowledge Conference (LAK 2026); April 27-May 01, 2026; Bergen, Norway CCS: Social and professional topics Student assessment CCS: Social a

Yufan Zhang, Jaromir Savelka, Seth Copen Goldstein, Michael Conway · 2026

Read Changes in Coding Behavior and Performance Since the Introduction of LLMs DOI: 10.1145/3785022.3785075 Conference: The 15th International Learning Analytics and Knowledge Conference (LAK 2026); April 27-May 01, 2026; Bergen, Norway CCS: Social and professional topics Student assessment CCS: Social a by Yufan Zhang, Jaromir Savelka, Seth Copen Goldstein, Michael Conway (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11835.txt.

Introduction

Yufan Zhang, Jaromir Savelka, Seth Copen Goldstein, and Michael Conway track how student coding changed once large language models spread. These tools can boost output. They can also hide how much a student has truly learned.

Their study looks at both work habits and scores in that shift. Teachers face a new task: to grade skill when code may come from a tool. The team frame what to watch and how to test fairly. The work is set for the LAK 2026 meeting on learning analytics. The study gives teachers a base for fair tests in a new era.

Catalog record

CREATORYufan Zhang, Jaromir Savelka, Seth Copen Goldstein, Michael Conway
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11835

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Changes in Coding Behavior and Performance Since the Introduction of LLMs DOI: 10.1145/3785022.3785075 Conference: The 15th International Learning Analytics and Knowledge Conference (LAK 2026); April 27-May 01, 2026; Bergen, Norway CCS: Social and professional topics Student assessment CCS: Social and professional topics Computing education CCS: Social and professional topics Computing education programs CCS: Human-centered computing Human computer interaction (HCI) CCS: Applied computing Education CCS: Software and its engineering Software prototyping Yufan Zhang, Jaromir Savelka, Seth Copen Goldstein, Michael Conway Abstract . The widespread availability of large language models (LLMs) has changed how students engage with coding and problem-solving. While these tools may increase student productivity, they also make it more difficult for instructors to assess students’ learning and effort. In this quasi-longitudinal study, we analyze five years of student source code submissions in a graduate-level cloud computing course, focusing on an assignment that remained unchanged and examining students’ behavior during the period spanning five semesters before the release of ChatGPT and five semesters after. Student coding behavior has changed significantly since Fall 2022. The length of their final submissions increased. Between consecutive submissions, average edit distances increased while average score improvement decreased, suggesting that both student productivity and learning have decreased after ChatGPT’s release. Additionally, there are statistically significant correlations between these behavioral changes and their overall performance. Although we cannot definitively attribute them to LLM misuse, they are consistent with our hypothesis that some students are over-reliant on LLMs, which is negatively affecting their learning outcomes. Our findings raise an alarm around the first generation of graduates in the age of LLMs, calling upon both educators and employers to reflect on their evaluation methods for genuine expertise and productivity. 1. Introduction Rapidly increasing capabilities of large language models (LLM) continue challenging existing practices in professional software development settings as well as in computer science education. The now well-established mode of interaction where an LLM is manually triggered to complete, create, or re-factor individual pieces of code is recently being replaced by workflows involving tools such as Claude Code 1 1 1 https://claude.com/product/claude-code or Gemini CLI 2 2 2 https://cloud.google.com/gemini/docs/codeassist/gemini-cli that are often referred to as agents. Although agentic frameworks handle increasingly more complex tasks such as planning, testing, and refactoring, human developers still play a crucial role in defining goals, guiding the process, and reviewing and fixing outcomes. At the extreme end, vibe coding is being used, where high-level natural-language directives orchestrate end-to-end software creation ( 15 ) . It appears that over a short period of several years, software development has transitioned from a state where most of the code is written by humans to one where the great majority of code is written by AI agents. At the same time, human supervision and intervention remain and will likely continue to remain indispensable. However, as there are increasingly fewer natural incentives for programmers to directly write and read code, a critical question emerges: How can we ensure that students and future software engineers develop and systematically hone their programming expertise? To even attempt to answer the question posed above we need to explore, quantitatively, how students are using LLMs in their work and how the use of an LLM is changing outcomes. Indeed, there has been a growing body of scholarship focused on how software developers and students use and perceive LLMs to assist them in carrying out their work. Although it is indisputable that students and software developers increasingly integrate LLM assistants into their workflows, empirical evidence on how this is happening, to what extent, and what effects it has remains scattered and limited. There are conflicting accounts ranging from how LLMs improve student learning ( 8 ; 16 ) to how LLMs harm students who over-rely on them ( 12 ) . There may also be a mismatch between students’ perceptions of increases in their productivity due to LLM usage and actual changes in their productivity. It has been shown that while experienced software developers may consider the use of LLM-powered assistants as boosting their productivity, in fact the effect may be quite opposite ( 2 ) . Hence, there is an urgent need for studies showing how the coding behaviors of students and software developers have been evolving in this rapidly transforming landscape and what effects these changes bring about. This paper analyzes students’ source code submissions in a graduate-level computer science course in cloud computing at Carnegie Mellon University in the United States, over 10 semesters between Fall 2020 and Spring 2025, i.e., precisely the period from several years before and several years after the introduction and mass popularization of LLM-powered coding assistants and dialog systems. The purpose of this paper is to understand how LLMs are impacting our students. We use code submissions and how they are made as a proxy to students’ learning behavior over the studied period. To that end, we employ and analyze a data set comprising 2,066 student submissions from the 10 semesters. The immediate insight that this study offers is that since the introduction of LLMs students approach to solving the task has changed significantly. They make more changes between consecutive submissions, they submit slightly more often, and they produce significantly more bloated solutions, as shown in Fig. 1. We show that these behavioral changes, though sometimes sufficient for completing programming tasks, may be translating into reduced fulfillment of the learning objectives while masking it under improved performance. Figure 1. Lines of code in final, perfect-score submissions increased significantly since Fall 2022. Note: the y-axis is not zero-indexed. Lines of code in final, perfect-score submissions increased significantly since Fall 2022. Note: the y-axis is not zero-indexed. Lines of code in final, perfect-score submissions increased significantly since Fall 2022. Note: the y-axis is not zero-indexed. We address the following research questions in our work: RQ1 How has coding behavior in individual assignments changed among students since LLM coding assistants and dialog systems became widely available? RQ2 Are changes in individual coding behavior associated with changes in learning efficiency and learning outcomes? To our best knowledge, this is the first comprehensive study that provides empirical evidence on changes in coding behavior, learning outcomes, and learning paths over 10 consecutive semesters of observations. We hope the quantitative analysis of student behavior presented in this paper will spur a deeper conversation about how our educational approach should shift in the age of AI. A useful lens for thinking about the needed transformation comes from the notion of centaurs in chess. The centaur, a team where humans and machines work together, outperforms either working on its own ( 3 ; 17 ) . The lesson from that domain is not simply that tools can make people faster, but that they fundamentally change what humans need to do, how they do it, and how we measure excellence. Software development is undergoing a similar shift. As AI agents take on larger portions of planning, scaffolding, testing, and implementation, the human role may be less about tactics (i.e., producing code) and more about strategy (i.e., defining abstractions, structuring complex systems, integrating components, and critically evaluating automated output). This has profound implications for project-based courses in computing education: our curricula must help students learn how to collaborate

Read the complete text