AI-Assisted Code Review as a Scaffold for Code Quality and Self-Regulated Learning: An Experience Report Conference: Accepted at ICSE-SEET ’26; ICSE 2026; SEET track
Eduardo Oliveira, Michael Fu, Patanamon Thongtanunam, Sonsoles López-Pernas, Mohammed Saqr · 2026
Read AI-Assisted Code Review as a Scaffold for Code Quality and Self-Regulated Learning: An Experience Report Conference: Accepted at ICSE-SEET ’26; ICSE 2026; SEET track by Eduardo Oliveira, Michael Fu, Patanamon Thongtanunam, Sonsoles López-Pernas, Mohammed Saqr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23251.txt.
Introduction
Oliveira and four colleagues study an LLM reviewer built into GitHub pull requests for capstone courses, where review is hard to scale. Across two cohorts of over 100 students, the 2024 group produced far more iterative activity. Failed AI attempts fell from 227 in 2023 to zero after tool and teaching refinements. Responsiveness stayed stable: about one-third of AI-reviewed pull requests drew a follow-up commit in each year. Students used structured AI comments to focus reviews and discuss quality, with guidance curbing over-reliance, yielding recommendations for responsible AI-assisted review.
Catalog record
| CREATOR | Eduardo Oliveira, Michael Fu, Patanamon Thongtanunam, Sonsoles López-Pernas, Mohammed Saqr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56757 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23251
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AI-Assisted Code Review as a Scaffold for Code Quality and Self-Regulated Learning: An Experience Report Conference: Accepted at ICSE-SEET ’26; ICSE 2026; SEET track Eduardo Oliveira, Michael Fu, Patanamon Thongtanunam, Sonsoles López-Pernas, Mohammed Saqr Abstract . Code review is central to software engineering education but hard to scale in capstones due to tight deadlines, uneven peer feedback, and limited prior experience. We investigate an LLM-as-reviewer integrated directly into GitHub pull requests (human-in-the-loop) across two cohorts (>100 students, 2023–2024). Using a mixed-methods design—GitHub data, reflective reports, and a targeted survey—we examine engagement and responsiveness as behavioural indicators of self-regulated learning processes. Quantitatively, the 2024 cohort produced more iterative activity (1176 vs. 581 PRs), while technical issues observed in 2023 (227 failed AI attempts) dropped to zero after tool and instructional refinements. Despite different adoption levels (teams using the tool: 93% vs. 50%), responsiveness was stable: 32% (2023) and 33% (2024) of successfully AI-reviewed PRs were followed by subsequent commits on the same PR. Qualitatively, students used the LLM’s structured comments to focus reviews and discuss code quality, while guidance reduced over-reliance. We contribute: (i) an in-workflow design for an AI reviewer that scaffolds learning while mitigating cognitive offloading; (ii) a repeated cross-sectional comparison across two cohorts in authentic settings; (iii) a mixed-methods analysis combining objective GitHub metrics with student self-reports; and (iv) evidence-based pedagogical recommendations for responsible, student-led AI-assisted review. 1. Introduction The proliferation of Large Language Models (LLMs) has catalysed a paradigm shift in software engineering practice. LLMs are no longer novelties but are rapidly integrating into professional development workflows, redefining tasks from code generation and debugging to documentation ( 3 ; 6 ) . This rapid integration presents a critical juncture for software engineering education. As educators, we are tasked with preparing students for an industry where collaboration with AI is becoming the norm. The central question is no longer whether to incorporate these tools, but how to do so in a manner that is pedagogically sound and enhances, rather than hinders, student learning. One of the most critical industry practices taught in software engineering curricula is code review. In professional settings, it is the foundation for quality assurance, knowledge sharing, and team collaboration ( 1 ; 20 ; 19 ) . However, translating this practice into an academic setting, particularly in high-stakes software engineering projects, is fraught with challenges. Students often lack the experience to provide insightful feedback, feel uncomfortable criticising their peers, and struggle with the time-consuming nature of manual reviews amidst pressing project deadlines ( 17 ; 2 ; 18 ; 21 ) . Consequently, peer code reviews in the classroom can be inconsistent, superficial, or neglected altogether, failing to impart the intended skills. To address these challenges, we present LLM-Reviewer, a custom tool we developed and integrated as an additional code reviewer within the existing pull-request workflow of a Master’s-level software engineering subject. Our tool is implemented as a repository-local automation (GitHub Action), allowing students to invoke it on-demand. Rather than introducing a new platform, it emits standard PR comments alongside human feedback. This design preserved the teams’ branching model, CI/CD pipeline, and sprint cadence, ensuring minimal disruption while capturing all interactions as first-class artefacts in the repository. The study spans two cohorts (2023–2024) and more than 100 students working with real industry clients on authentic projects. Our focus is model-agnostic: we examine the pedagogical and behavioural effects of positioning an LLM inside the established review loop, not the idiosyncrasies of any particular product. We frame our intervention not merely as an automation tool, but as a pedagogical scaffold for fostering Self-Regulated Learning (SRL). This LLM-as-reviewer complements conventional linters and static checkers by providing structured, rubric-aligned natural-language feedback (e.g., rationale, documentation, and design considerations) while keeping humans in the loop for judgement and decisions. By providing immediate, structured feedback designed to prompt reflection—rather than supplying definitive answers—our LLM-Reviewer tool aimed to help students better monitor their work, evaluate its quality, and reflect on their coding practices, thereby fostering the metacognitive skills essential for lifelong learning. To investigate the impact of our LLM-Reviewer tool for SRL and critical thinking, we use a mixed-methods analysis of repository data, surveys, and detailed student reflections. This study examines student engagement with the tool, their perceptions of its utility, and the broader challenges of its integration. In doing so, this work makes four key contributions to software engineering education: (i) A novel design for an AI review tool that scaffolds learning while mitigating the risk of cognitive offloading; (ii) a detailed repeated cross-sectional comparison between two cohorts on the same subject with a consistent method on the integration of our LLM-Reviewer tool into authentic software engineering settings, offering insights into replicability and stability of our findings; (iii) a mixed methods analysis of student engagement, combining objective behavioral metrics with subjective self-reports and survey responses; and (iv) a set of evidence-based pedagogical recommendations for educators seeking to navigate the new landscape of AI-assisted software engineering. 2. Background & Related Work 2.1. Code review in education In modern software engineering, the pull request (PR) serves as the central mechanism for collaborative development. It is a formal proposal to merge a set of changes—such as new features, bug fixes, or improvements—from a development branch into a main code base ( 18 ) . The core of the PR process is code review, a practice widely acknowledged as a basis of quality assurance in professional settings. Industry-based studies have long established that code review is critical for detecting defects, enforcing team-specific coding standards, sharing knowledge across a team, and improving the long-term quality and maintainability of software projects ( 1 ; 17 ) . Recognising its industrial importance, software engineering educators have increasingly integrated code review and PR workflows into their curricula, particularly in project-based and software-based capstone courses ( 18 ) . The pedagogical goal extends beyond mere defect detection; it is to provide students with authentic experience in critical analysis, collaborative development, and the socio-technical skill of giving and receiving constructive feedback ( 17 ) . However, translating this practice effectively into the classroom is challenging ( 9 ; 10 ) . Studies consistently report that students often lack the technical expertise to provide insightful feedback, leading to reviews that can be superficial or inconsistent ( 2 ; 1 ) . Furthermore, significant socio-cognitive barriers exist; students frequently feel uncomfortable critiquing their peers’ work due to concerns about fairness, bias, or social dynamics, a contrast to the structured expectations of a professional environment ( 17 ) . Compounded by the time-consuming nature of manual reviews amidst the pressure of project deadlines, the result is that peer code review in academic settings is often neglected or fails to deliver on its pedagogical promise, creating a clear need for alternative or supplementary approaches. 2.2. AI-driven feedback as a scaffold for fostering SRL The recent emergence of LLMs has introduced a powerful new category of tools into the software engineering landscape. General-purpose LLMs and specialised tools such as GitHub Copilot are now capable of a wide range of development tasks, including code generation, explanation, and debugging ( 2 ; 18 ) . This has led to the development of AI tools and agents specifically designed