Will Your Next Pair Programming Partner Be Human? An Empirical Evaluation of Generative AI as a Collaborative Teammate in a Semester-Long Classroom Setting Conference: Proceedings of the Twelfth ACM Conference on Learning @ Scale; July 21–23, 2025; Palermo, Italy Proceedings of the Twelfth ACM Confe
Wenhan Lyu, Yimeng Wang, Yifan Sun, Yixuan Zhang · 2025
Read Will Your Next Pair Programming Partner Be Human? An Empirical Evaluation of Generative AI as a Collaborative Teammate in a Semester-Long Classroom Setting Conference: Proceedings of the Twelfth ACM Conference on Learning @ Scale; July 21–23, 2025; Palermo, Italy Proceedings of the Twelfth ACM Confe by Wenhan Lyu, Yimeng Wang, Yifan Sun, Yixuan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08119.txt.
Introduction
Wenhan Lyu and co-authors report in 2025 a term-long class study on coding with a smart chat tool as a pair partner. Pair coding means two people share one task and swap roles. The team asks how the new tools change grades, teamwork, and how students learn.
Thirty nine undergraduates did six class tasks in three ways: pairs alone, pairs with the tools, and solo with the tools. Pairs with tool help scored the best, while solo work with tools scored the lowest. Views of the tools grew more warm after use. Likes and dislikes tracked how much help students felt the tools gave.
Catalog record
| CREATOR | Wenhan Lyu, Yimeng Wang, Yifan Sun, Yixuan Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08119
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Will Your Next Pair Programming Partner Be Human? An Empirical Evaluation of Generative AI as a Collaborative Teammate in a Semester-Long Classroom Setting Conference: Proceedings of the Twelfth ACM Conference on Learning @ Scale; July 21–23, 2025; Palermo, Italy Proceedings of the Twelfth ACM Conference on Learning @ Scale(L@S ’25), July 21–23, 2025, Palermo, Italy DOI: 10.1145/3698205.3729544 ISBN: 979-8-4007-1291-3/2025/07 CCS: Human-centered computing Human computer interaction (HCI) CCS: Human-centered computing Collaborative and social computing Wenhan Lyu, Yimeng Wang, Yifan Sun, Yixuan Zhang Abstract . Generative AI (GenAI), especially Large Language Models (LLMs), is rapidly reshaping both programming workflows and computer science education. Many programmers now incorporate GenAI tools into their workflows, including for collaborative coding tasks such as pair programming. While prior research has demonstrated the benefits of traditional pair programming and begun to explore GenAI-assisted coding, the role of LLM-based tools as collaborators in pair programming remains underexamined. In this work, we conducted a mixed-methods study with 39 undergraduate students to examine how GenAI influences collaboration, learning, and performance in pair programming. Specifically, students completed six in-class assignments under three conditions: Traditional P air P rogramming ( PP ), P air Programming with Gen AI ( PAI ), and S olo Programming with Gen AI ( SAI ). They used both LLM-based inline completion tools (e.g., GitHub Copilot) and LLM-based conversational tools (e.g., ChatGPT). Our results show that students in the PAI condition achieved the highest assignment scores, whereas those in the SAI condition attained the lowest. Additionally, students’ attitudes toward LLMs’ programming capabilities improved significantly after collaborating with LLM-based tools, and preferences were largely shaped by the perceived usefulness for completing assignments and learning programming skills, as well as the quality of collaboration. Our qualitative findings further reveal that while students appreciated LLM-based tools as valuable pair programming partners, they also identified limitations (e.g., contextual constraints and possibly outdated knowledge bases), and had different expectations compared to human teammates. Students in our study primarily relied on LLM-based tools for syntax clarification and conceptual guidance, while turning to human partners for idea exchanges. Our study provides one of the first empirical evaluations of GenAI as a pair programming collaborator through a comparison of three conditions ( PP , PAI , and SAI ). We also discuss the design implications and pedagogical considerations for future GenAI-assisted pair programming approaches. † † cc-license: by 1. Introduction Recent advancements in Generative AI (GenAI) have ushered in a transformative area in programming ( Jo 2023 ; Sun et al. 2022 ; Ebert and Louridas 2023 ; Bull and Kharrufa 2024 ) , driven by breakthroughs in Large Language Models (LLMs). LLM-based applications (e.g., ChatGPT ( OpenAI [n. d.] ) , GitHub Copilot ( GitHub [n. d.] ) ) provide capabilities such as code generating ( Kazemitabaar et al. 2023 ; Ouyang et al. 2025 ; Sakib et al. 2024 ) and debugging assistance ( Sakib et al. 2024 ; Kang et al. 2025 ) , delivered through forms like chatbots ( Dam et al. 2024 ; OpenAI [n. d.] ) and inline completions ( GitHub [n. d.] ; Husein et al. 2024 ) . LLM-based innovations with human-AI collaboration potentials are reshaping programming workflows and posing new challenges in computer science (CS) education ( Denny et al. 2024a ; Prather et al. 2024 ; Phung et al. 2023 ) , including traditional human-human collaborative approaches such as pair programming. Pair programming ( Beck 2000 ) has long been established as a powerful collaborative strategy in both CS education and industry practices ( Hannay et al. 2009 ; Hanks et al. 2011 ) , recognized for its effectiveness in enhancing problem-solving skills ( Williams and Upchurch 2001 ) and code quality ( Hulkko and Abrahamsson 2005 ) . Traditionally, pair programming has relied on the dynamic interplay between two human collaborators, where one performs coding and implementations while the other provides assistance like debugging and reviewing, each fulfilling distinct roles that complement the other ( Hannay et al. 2009 ; Hanks et al. 2011 ) . However, the rapid advancement of GenAI introduces a new paradigm shift from an auxiliary tool to a collaborator, providing insights comparable to those of a human partner in pair programming. As such, understanding GenAI’s role in pair programming and its impact on collaboration in education is key to assessing its effectiveness as a learning and programming partner, which might further shape its potential to enhance human interaction in CS education. While prior research highlights the benefits of traditional pair programming ( Williams and Upchurch 2001 ; Hulkko and Abrahamsson 2005 ) and explores GenAI-assisted coding ( Lyu et al. 2024 ; Xue et al. 2024 ) , the prospect of treating LLM-based tools as collaborators rather than mere aids remains largely unexamined ( Ma et al. 2023 ) . The underexplored potential of GenAI to fully engage in the collaborative dynamics of pair programming underscores the need for classroom deployments that assess GenAI’s ability to either complement or replace human programmers. Furthermore, understanding how the integration of GenAI into pair programming processes might influence students’ attitudes towards GenAI is crucial as it can inform effective collaboration strategies and enhance overall educational outcomes. The following research questions (RQs) guide our work: RQ1. In what ways do collaborations with LLM-based tools as pair programming partners influence students’ attitudes towards GenAI on programming? RQ2. How does integrating LLM-based tools as collaborators in pair programming affect student learning outcomes? and RQ3. What differences emerge in students’ experiences and preferences when comparing pair programming with LLM-based tools versus traditional human partners? To answer these questions, we conducted a mixed-method study from September to December 2024 with 39 students in an undergraduate CS course. Throughout the semester, participants completed six in-class assignments under three pair programming conditions: Traditional P air P rogramming ( PP ), where two human programmers collaborate without GenAI; P air Programming with Gen AI ( PAI ), in which two human programmers work alongside LLM-based collaborators; and S olo Programming with Gen AI ( SAI ), where a single programmer is paired with LLM-based collaborators. We collected assignment scores and surveyed students’ attitudes towards LLM-based tools in the context of pair programming (including both structured survey questions and open-ended responses), as well as students’ reflection reports on their collaborations with LLM-based tools. Our quantitative results show 1) students’ overall attitudes towards LLM-based tools in programming improved significantly after working with LLM-based tools as pair programming collaborators over time, 2) students’ assignment score was highest in the PAI and lowest in the SAI condition, and 3) both the collaborative experience and the perceived usefulness influenced students’ preferences for the different pair programming conditions. Our qualitative analysis further revealed that 1) in general, students considered LLM-based tools as useful collaborators in pair programming, though they noted certain limitations, such as contextual constraints and outdated knowledge bases, 2) students expressed varying expectations regarding the roles of human collaborators versus LLM-based collaborators during pair programming, where treating LLM-based collaborators more as complements for syntax and concepts assistance while seeking idea exchange with human collaborators, and 3) students demonstrated different preferences for interaction modalities, with some favoring chatbots and others preferring inline completions as their LLM-based