Unveiling the Role of ChatGPT in Software Development: Insights from Developer-ChatGPT Interactions on GitHub
Ruiyin Li, Peng Liang, Yifei Wang, Yangxiao Cai, Weisong Sun, Zengyang Li · 2025
Read Unveiling the Role of ChatGPT in Software Development: Insights from Developer-ChatGPT Interactions on GitHub by Ruiyin Li, Peng Liang, Yifei Wang, Yangxiao Cai, Weisong Sun, Zengyang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03901.txt.
Introduction
Ruiyin Li and co-authors study how programmers actually use ChatGPT, in this 2025 paper. They gathered more than twenty-five hundred shared ChatGPT conversation links posted on GitHub over a year, and built a dataset called DevChat. From those real chats, they sorted why developers share the talks and which tasks fill them.
Most exchanges were short, one to three turns, and sharply task focused. Developers shared talks mainly to hand off work, fix problems, or learn. Coding and code upkeep dominated, with code writing and code tuning the two busiest tasks. The authors map thirty-nine fine-grained tasks the tool supports in practice.
Catalog record
| CREATOR | Ruiyin Li, Peng Liang, Yifei Wang, Yangxiao Cai, Weisong Sun, Zengyang Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03901
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unveiling the Role of ChatGPT in Software Development: Insights from Developer-ChatGPT Interactions on GitHub Ruiyin Li, Peng Liang, Yifei Wang, Yangxiao Cai, Weisong Sun, Zengyang Li Abstract . The advent of Large Language Models (LLMs) has introduced a new paradigm in Software Engineering (SE), with generative AI tools like ChatGPT gaining widespread adoption among developers. While ChatGPT’s potential has been extensively discussed, empirical evidence about how developers actually use LLMs’ assistance in real-world practices remains limited. To bridge this gap, we conducted a large-scale empirical analysis of ChatGPT usage on GitHub, and we presented DevChat, a curated dataset of 2,547 publicly shared ChatGPT conversation links collected from GitHub between May 2023 and June 2024. Through comprehensively analyzing DevChat, we explored the characteristics of developer–ChatGPT interaction patterns and identified five key categories of developers’ purposes of using developer-ChatGPT conversations during software development. Additionally, we investigated the dominant development-related activities in which ChatGPT is used, and presented a mapping framework that links GitHub data sources, development-related activities, and SE tasks. The findings show that interactions are typically short and task-focused (most are 1-3 turns); developers share conversations mainly to delegate tasks, resolve problems, and acquire knowledge, revealing five purpose categories; ChatGPT is most frequently engaged for Software Implementation and Maintenance & Evolution ; we identified 39 fine-grained SE tasks supported by ChatGPT, with Code Generation & Completion as well as Code modification & Optimization being the most prominent. Our study offers a comprehensive mapping of ChatGPT’s applications in real-world software development scenarios and provides a foundation for understanding LLMs’ practical roles in software development. Journal: TOSEM Volume: 0 0 0 0 CCS: Software and its engineering Software development techniques 1. Introduction The emergence of Large Language Models (LLMs) has fundamentally reshaped the landscape of Software Engineering (SE) practices ( Hou et al., 2024 ) . They introduce a novel paradigm for software development, offering capabilities that can potentially revolutionize the field. As one of the most typical and representative LLM-based tools, ChatGPT ( OpenAI, ChatGPT, 2025 ) , launched by OpenAI in November 2022, has sparked widespread interest within both academia and industry, leading to numerous studies exploring its applicability in software development, such as code generation and refinement ( Sobania et al., 2023 ; Guo et al., 2024 ; Li et al., 2023 ) , code summarization and comment generation ( Sun et al., 2025 ; Su and McMillan, 2024 ) , and program repair ( Zhang et al., 2023 ) . Although prior research has demonstrated ChatGPT’s capabilities and evaluated its performance on specific tasks, there remains limited empirical understanding of how developers actually use ChatGPT in real-world software development workflows, and for what purposes and tasks it is leveraged in daily practice. A key challenge in studying real-world ChatGPT usage is the lack of observable traces that reflect developers’ authentic practices. In May 2023, OpenAI released a new feature allowing users to share links within their ChatGPT interactions ( ChatGPT Shared Links FAQ, 2025 ) . This feature promotes the sharing of knowledge generated by ChatGPT and enhances team collaboration through shared developer-ChatGPT interactions. Developers create a unique record of ChatGPT-assisted development by embedding these links in GitHub workflows. As a result, this feature has further boosted ChatGPT’s popularity among developers, surpassing other LLMs in developers’ communities. As evidenced by our trial search, before selecting ChatGPT as our target LLM, we conducted a preliminary search on GitHub for other LLMs that support shared conversation links, such as Gemini ( Google, Gemini, 2025 ) , and found significantly fewer shared links. Consequently, we decided to use ChatGPT as the focus of this study, which is the most popular and widely used LLM on GitHub. Recent work has begun to explore such developer–ChatGPT interactions. For example, Xiao et al . ( Xiao et al., 2024 ) shared a dataset of developer-ChatGPT conversations used on GitHub, collected from July to October 2023, which ignited interest in studying ChatGPT’s practical use. However, due to its relatively small size and temporal coverage, the dataset is not sufficient to support our comprehensive understanding of the prevalence and usability of ChatGPT in software development. Compared to the developer survey of ChatGPT usage conducted by Vaillant et al . ( Vaillant et al., 2024 ) , our work provides empirical evidence of actual usage through observable ChatGPT links that developers proactively share. These links constitute concrete artifacts of ChatGPT usage, reflecting instances where developers considered the interactions sufficiently valuable to reference or reuse within development workflows. As such, our work offers complementary insights to perception- and recall-based survey studies by grounding the analysis in actually occurring usage data. We acknowledge that shared ChatGPT links represent only a subset of ChatGPT’s overall usage in practice. Nevertheless, the links offer a valuable observation window into real-world developers’ practices, capturing interactions that developers deem noteworthy or relevant enough to externalize. While this study does not aim to assess the direct impact of these shared conversations on development outcomes, it focuses on characterizing how ChatGPT is leveraged in developers’ workflows through these observable usage traces. In this work, we mined and analyzed such shared and available ChatGPT links on GitHub to investigate the current landscape and patterns of ChatGPT usage in software development. For brevity, we henceforth refer to “publicly shared ChatGPT usage” simply as “the usage of ChatGPT” or “ChatGPT usage” in this paper. Additionally, compared to previous studies published in the Mining Challenge track of MSR 2024 ( Mining Challenge Track of MSR 2024, 2024 ) that investigated the developer-ChatGPT interactions from certain perspectives (e.g., code generation, prompt interactions), our work comprehensively analyzed the ChatGPT usage during the software development process on GitHub through the shared and available ChatGPT links, including characterizing developer-ChatGPT interactions , uncovering developers’ motivations , analyzing the development-related activities developers use ChatGPT for, mapping GitHub data sources, development-related activities, and concrete SE tasks performed with ChatGPT’s assistance. The integration of ChatGPT into software development workflows is still in its early stages, but it has already shown great promise in improving developer productivity and code quality ( Vaillant et al., 2024 ) . Therefore, our goal is to investigate how developers utilize ChatGPT in their daily routines to better anticipate its broader impact on the SE field. To this end, we conducted a large-scale empirical investigation into shared ChatGPT links, offering insights into LLM-assisted software development. Specifically, we presented DevChat , a curated dataset of 2,547 unique shared ChatGPT conversations collected from GitHub between May 2023 and June 2024. These conversations are drawn from five major GitHub data sources (i.e., Code, Issues, Commits, Pull Requests, and Discussions), capturing a diverse range of development scenarios. Leveraging DevChat, we conducted an in-depth analysis of how developers utilize ChatGPT during software development. Our analysis reveals several key findings : Developer–ChatGPT interactions are typically involving short, low-turn conversations (most are 1-3 turns); Developers’ purposes of using ChatGPT links on GitHub were identified and categorized into five categories, with Task Delegation being the dominant purpose; We categorized the development-related activities within the shared ChatGPT conversations into two primary groups: Lifecycle Activities and Supporting Activities , with Software Implementation , Software Maintenance and Evolution emerged as the dominant activities; By systematically mapping