GitHub’s Copilot Code Review: Can AI Spot Security Flaws Before You Commit?
Amena Amro, Manar H. Alalfi · 2025
Read GitHub’s Copilot Code Review: Can AI Spot Security Flaws Before You Commit? by Amena Amro, Manar H. Alalfi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13650.txt.
Introduction
Amena Amro and Manar H. Alalfi test GitHub Copilot's code review feature in a 2025 software security study. The tool promises to comment on code before it is committed, and the authors ask whether it can actually spot security flaws, a skill developers increasingly assume such assistants have.
They ran the reviewer over a curated set of labeled vulnerable code samples drawn from open-source projects across languages and domains. The reviewer often missed critical flaws such as SQL injection, cross-site scripting, and insecure deserialization. Its comments mostly flagged low-severity matters like style and typos instead. The authors conclude that dedicated security tools and manual audits remain necessary, whatever the marketing suggests.
Catalog record
| CREATOR | Amena Amro, Manar H. Alalfi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-13650
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GitHub’s Copilot Code Review: Can AI Spot Security Flaws Before You Commit? Amena Amro, Manar H. Alalfi Abstract As software development practices increasingly adopt AI-powered tools, ensuring that such tools can support secure coding has become critical. This study evaluates the effectiveness of GitHub Copilot’s recently introduced code review feature in detecting security vulnerabilities. Using a curated set of labeled vulnerable code samples drawn from diverse open-source projects spanning multiple programming languages and application domains, we systematically assessed Copilot’s ability to identify and provide feedback on common security flaws. Contrary to expectations, our results reveal that Copilot’s code review frequently fails to detect critical vulnerabilities such as SQL injection, cross-site scripting (XSS), and insecure deserialization. Instead, its feedback primarily addresses low-severity issues, such as coding style and typographical errors. These findings expose a significant gap between the perceived capabilities of AI-assisted code review and its actual effectiveness in supporting secure development practices. Our results highlight the continued necessity of dedicated security tools and manual code audits to ensure robust software security. I Introduction In an era where software security is paramount, the integration of AI-powered tools such as GitHub Copilot is transforming the software development process. While Copilot has demonstrated strong capabilities in code generation and developer assistance, its effectiveness in identifying and mitigating security vulnerabilities remains uncertain, particularly as its underlying large language models (LLMs) are continuously evolving [ 6 ] . This study investigates the performance of Copilot’s newly introduced code review feature, which, as of February 2025, remains in public preview. Our goal is to assess how well this feature can detect known security flaws in the source code. To evaluate Copilot’s security review capabilities, we employ labeled vulnerable code samples drawn from a range of publicly available datasets [ 2 , 3 , 4 , 11 , 5 ] . These datasets encompass a diverse set of codebases, including web applications, mobile applications, and APIs implemented in multiple programming languages [ 3 ] . One dataset [ 4 ] offers an expanding suite of test cases with documented weaknesses, while another [ 11 ] focuses on known vulnerabilities in widely used libraries and packages. In addition, a vulnerable web service and API [ 5 ] is included to simulate realistic enterprise use cases. All datasets provide ground-truth vulnerability labels, which we use as a benchmark to evaluate Copilot’s ability to correctly identify and annotate security issues during code review. Our evaluation focuses on two key dimensions: (1) the accuracy of Copilot’s code review in detecting security vulnerabilities and (2) the clarity and usefulness of its feedback in helping developers understand and resolve these issues. By systematically analyzing Copilot’s performance in these areas, we aim to uncover both its strengths and limitations as a security review assistant. Ultimately, our findings contribute to a deeper understanding of the current capabilities of AI-assisted secure coding tools and inform directions for future improvement. II Background and Related Work GitHub Copilot is an AI-powered code completion and synthesis tool developed by GitHub in collaboration with OpenAI. It was originally powered by Codex, a large language model (LLM) trained on a large corpus of public code and natural language. However in 2023 and beyond, GitHub Copilot uses more advanced models such as GPT-4 [ 1 ] . Originally released to assist developers in writing code by offering inline suggestions, Copilot has recently expanded its functionality to include automated code review. As of early 2025, the Copilot Code Review feature provides natural language feedback on code changes submitted through pull requests. This feature aims to emulate the role of a peer reviewer by identifying potential issues, including security concerns, and suggesting improvements. Unlike traditional rule-based static analyzers, Copilot’s code review capability is powered by deep learning models trained on vast corpora of publicly available code and associated documentation. Consequently, it does not follow predefined rules or leverage formal program semantics; instead, it relies on learned statistical correlations and contextual patterns [ 6 ] . This raises important questions regarding its reliability, particularly in detecting complex or subtle security flaws that may not follow syntactic norms. Security concerns surrounding LLM-generated code have been well-documented. Perry et al. [ 8 ] conducted a controlled user study and found that developers using Copilot were more likely to submit insecure solutions compared to those who coded without AI assistance. Alarmingly, these developers were often more confident in their submissions, despite the presence of vulnerabilities. Tihanyi et al. [ 9 ] conducted a large-scale empirical study involving over 330,000 C programs generated by multiple LLMs and found that 62% of the outputs contained at least one security vulnerability. Similarly, Wang et al. [ 10 ] evaluated LLMs on secure code generation tasks and found that while models performed reasonably on code synthesis, they were significantly less effective in repairing scenarios where existing vulnerabilities must be corrected. The limitations of LLMs in security-focused tasks raise the need to compare them with established static analysis tools. Tools like GitHub’s CodeQL rely on semantic-aware, rule-based detection mechanisms that offer consistent and explainable diagnostics. CodeQL operates by constructing an abstract semantic graph of the code and queries it using a declarative logic-based language to identify common vulnerability patterns. While precise and customizable, CodeQL and similar tools are typically language-specific and may struggle with code written in unconventional styles. In contrast, LLM-based reviewers such as Copilot are language-agnostic and capable of offering human-like commentary across a broad range of contexts. However, their probabilistic nature leads to inconsistencies and false positives or negatives, especially when facing novel or obfuscated vulnerability patterns [ 6 ] . Understanding how Copilot’s code review compares to traditional tools like CodeQL is essential to evaluate its utility for secure software development. A growing body of related work explores LLMs’ ability not only to generate secure code, but also to assist in code repair and review. Pearce et al. [ 6 ] found that approximately 40% of Copilot-generated programs were vulnerable, even when evaluated against GitHub’s own CodeQL engine. In another relevant study, Pearce et al. [ 7 ] explored the capabilities of LLMs to fix known security vulnerabilities in a zero-shot setting and found that while some models produced functional patches, success was inconsistent and highly dependent on prompt quality. These studies emphasize the potential value and current limitations of LLMs in supporting secure software engineering workflows. In light of this evidence, our work focuses specifically on evaluating GitHub Copilot’s Code Review feature: its ability to detect known vulnerabilities in real-world, vulnerable codebases, and the nature of the feedback it provides. To our knowledge, this is one of the first empirical studies evaluating this feature across multiple datasets and application domains. III Methodology III-A Research Objective The primary objective of this study is to empirically evaluate the effectiveness of GitHub Copilot’s newly introduced code review feature in identifying security vulnerabilities in source code. Specifically, we evaluate Copilot’s ability to (1) detect known security flaws and (2) provide actionable, contextually appropriate feedback for remediation. As a baseline, we compare Copilot’s performance against GitHub’s CodeQL static analysis engine, focusing on both the detection accuracy and the quality of review comments. III-B Evaluation Design To conduct this evaluation, we selected a diverse