The Impact of Generative AI on Code Expertise Models: An Exploratory Study Conference: Brazilian Symposium on Software Engineering; September 22-26, 2025; Recife, PE
Otávio Cury, Guilherme Avelino · 2025
Read The Impact of Generative AI on Code Expertise Models: An Exploratory Study Conference: Brazilian Symposium on Software Engineering; September 22-26, 2025; Recife, PE by Otávio Cury, Guilherme Avelino (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08160.txt.
Introduction
Otávio Cury and Guilherme Avelino asked whether AI code generation corrupts models of developer expertise in a 2025 study. Teams identify experts by mining who wrote and knows which code, feeding truck factor estimates of how many departures would strand a project. If much new code is actually machine-written, the knowledge a repository attributes to a developer may overstate what they truly understand, and those metrics could quietly rot.
The authors gathered statistics on ChatGPT-written code entering GitHub projects and simulated expertise models under varying shares of AI contribution. Most scenarios showed measurable distortion of the knowledge model and its truck factor output. Expertise metrics, they conclude, need rethinking as generated code spreads.
Catalog record
| CREATOR | Otávio Cury, Guilherme Avelino |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08160
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Impact of Generative AI on Code Expertise Models: An Exploratory Study Conference: Brazilian Symposium on Software Engineering; September 22-26, 2025; Recife, PE Otávio Cury, Guilherme Avelino Abstract . Generative Artificial Intelligence (GenAI) tools for source code generation have significantly boosted productivity in software development. However, they also raise concerns, particularly the risk that developers may rely heavily on these tools, reducing their understanding of the generated code. We hypothesize that this loss of understanding may be reflected in source code knowledge models, which are used to identify developer expertise. In this work, we present an exploratory analysis of how a knowledge model and a Truck Factor algorithm built upon it can be affected by GenAI usage. To investigate this, we collected statistical data on the integration of ChatGPT-generated code into GitHub projects and simulated various scenarios by adjusting the degree of GenAI contribution. Our findings reveal that most scenarios led to measurable impacts, indicating the sensitivity of current expertise metrics. This suggests that as GenAI becomes more integrated into development workflows, the reliability of such metrics may decrease. 1. Introduction In recent years, Generative Artificial Intelligence (GenAI) has attracted considerable attention from the Software Engineering community, revealing research gaps across multiple areas ( Nguyen-Duc et al. 2023 ) . This growing interest is justified by GenAI’s potential to transform a range of domains ( Gozalo-Brizuela and Garrido-Merchán 2023 ) . In software development, the increasing integration of GenAI into development environments has brought numerous benefits. Tools such as OpenAI’s ChatGPT 1 1 1 https://chatgpt.com and GitHub Copilot 2 2 2 https://github.com/features/copilot have contributed to increased productivity and enhanced software quality ( Dohmke et al. 2023 ; Ziegler et al. 2024 ) . Despite these advantages, researchers have raised concerns related to adopting such technologies ( Ernst and Bavota 2022 ) . One significant drawback is the risk of developer over-reliance on GenAI tools, where users may accept generated code without fully understanding it ( Zhang et al. 2023 ; Yilmaz and Yilmaz 2023 ) . Such behavior can undermine problem-solving skills and hinder a deeper comprehension of the codebase ( Prather et al. 2023 ) . Given this, we argue that models used to estimate developer expertise should reflect the potential impact of GenAI-assisted code generation on code understanding. Typically based on authorship data of Version Control Systems , these models estimate a developer’s knowledge of the code and are applied across various stages of the software development ( Hannebauer et al. 2016 ; Cury et al. 2022 ) . They also play a key role in knowledge loss mitigation, being used in metrics such as the Truck Factor , which estimates the concentration of knowledge within a development team, a well-established line of research ( Cury et al. 2024 ; Avelino et al. [n. d.] ; Jabrayilzade et al. 2022 ) . Building on this premise, this paper investigates how developers’ use of GenAI tools may impact source code expertise identification models. For this investigation, we searched for ChatGPT shared links embedded in source code files from GitHub projects. Using these links, we retrieved the corresponding conversations and identified the extent to which GenAI-generated code had been integrated into the project files. Based on statistical evidence of this integration, we simulated different scenarios to assess the potential impact on expertise identification models and a widely used Truck Factor algorithm, by attributing a portion of developer authorship to GenAI. This study is guided by the following overarching question: How does the use of GenAI tools for source code generation affect the accuracy and reliability of models that identify developer expertise in source code? The key contributions of this study are as follows: 1) Quantitative insights into how code generated by ChatGPT is integrated into open-source projects; 2) A comprehensive analysis of how this integration may impact expertise models and their applications, such as knowledge concentration metrics; 3) A publicly available dataset containing information on the integration of GenAI-generated code in open-source repositories. This work is organized as follows: Section 2 introduces key concepts. Section 3 reviews related studies on the relationship between GenAI tools and source code knowledge. Section 4 describes the data collection process used in this study. Section 5 presents the results, Section 6 discusses them, and Section 7 concludes the paper. 2. Background 2.1. Code Knowledge Models Source code knowledge models are designed to identify developer expertise within software projects. This information supports a range of activities, including task assignment and bug fixing, and also assists project managers in monitoring knowledge concentration within the codebase ( Ferreira et al. 2017 ; Robillard 2021 ) . Most existing approaches rely on development history stored in Version Control Systems . For example, the Degree of Knowledge model, proposed by Fritz et al., combines two types of information: the developer’s authorship of a file captured by the Degree of Authorship (DOA), and the number of interactions the developer has had with that file ( Fritz et al. 2014 ) . 2.1.1. Degree of Expertise In a previous study, we proposed the Degree of Expertise (DOE), a knowledge model that uses four variables from development history to measure a developer’s knowledge on a source code file ( Cury et al. 2022 ) . Unlike existing models in the literature, DOE combines fine-grained measures of change, authorship, recency of modification, and file size to achieve greater precision in knowledge estimation. The DOE model showed superior performance in identifying file experts ( Cury et al. 2022 ) , and was applied within a Truck Factor algorithm ( Cury et al. 2024 ; Cury and Avelino 2024 ) . The DOE of a developer d in version v of file f is calculated using Equation 1 . \displaystyle\textbf{DOE(d, f(v))}= \displaystyle\;5.28223+0.23173\cdot\ln(1+\textbf{Adds}^{d,f(v)}) \displaystyle+0.36151\cdot(\textbf{FA}^{f})-0.28761\cdot\ln(\textbf{Size}^{f(v)}) (1) \displaystyle-0.19421\cdot\ln(1+\textbf{NumDays}^{d,f(v)}) Where, Adds : number of lines added by developers d on file f ; FA : boolean if developer d is the creator of the file f ; Size : number of lines of code (LOC) of the file f ; NumDays : number of days since the last commit of a developer d on file f . In this work, we examine how the integration of GenAI-generated code impacts this knowledge model. The rationale behind selecting this particular model is further detailed on Section 4 . 2.2. Truck Factor Algorithm The Truck Factor measures how many developers would need to leave a software project for it to be critically affected ( Jabrayilzade et al. 2022 ) . This metric helps identify knowledge concentration risks and has been widely studied in the literature ( Jabrayilzade et al. 2022 ; Ferreira et al. 2017 ; Avelino et al. 2016 ; Cury et al. 2024 ) . Among the proposed methods for estimating Truck Factor, Avelino’s algorithm ( Avelino et al. 2016 ) is notable for its strong performance in comparison studies ( Ferreira et al. 2017 ) and for being adopted in subsequent validation studies ( Calefato et al. 2022 ; Almarimi et al. 2021 ) . Avelino’s algorithm estimates the Truck Factor using a strategy based on developer authorship. It identifies file experts through the Degree of Authorship (DOA) model and iteratively removes the developer who is the expert for