From Code Generation to Software Testing: AI Copilot with Context-Based RAG
Yuchen Wang, Shangxin Guo, Chee Wei Tan · 2025
Read From Code Generation to Software Testing: AI Copilot with Context-Based RAG by Yuchen Wang, Shangxin Guo, Chee Wei Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01866.txt.
Introduction
Yuchen Wang, Shangxin Guo, and Chee Wei Tan link code writing and code testing in one AI helper in a 2025 paper. Big software teams ship code fast, and tests often lag that pace. Bugs then slip through, and finding them later costs far more than catching them at the desk.
Their helper reads the whole code base for context while it helps write and test code. When code changes, the tests shift with it, so the two stay in step. In trials, bug spotting rose by 31.2 percent and key test cover rose by 12.6 percent. Users also took up its hints more often than a base tool's hints.
Catalog record
| CREATOR | Yuchen Wang, Shangxin Guo, Chee Wei Tan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-01866
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Code Generation to Software Testing: AI Copilot with Context-Based RAG Yuchen Wang, Shangxin Guo, Chee Wei Tan Abstract The rapid pace of large-scale software development places increasing demands on traditional testing methodologies, often leading to bottlenecks in efficiency, accuracy, and coverage. We propose a novel perspective on software testing by positing “bug detection" and “coding with fewer bugs" as two interconnected problems that share a common goal—reducing bugs with limited resources. We extend our previous work on AI-assisted programming, which supports code auto-completion and chatbot-powered Q&A, to the realm of software testing. We introduce “Copilot for Testing," an automated testing system that synchronizes bug detection with codebase updates, leveraging context-based Retrieval Augmented Generation (RAG) to enhance the capabilities of large language models (LLMs). Our evaluation demonstrates a 31.2% improvement in bug detection accuracy, a 12.6% increase in critical test coverage, and a 10.5% higher user acceptance rate, highlighting the transformative potential of AI-driven technologies in modern software development practices. Keywords AI-assisted Testing \cdot Retrieval-Augmented Generation (RAG) \cdot Software Quality Assurance \cdot Bug Detection \cdot Automated Software Testing 1 Introduction In the evolving landscape of software development, the rapid pace of innovation and the increasing complexity of systems demand equally sophisticated tools to ensure reliability and efficiency. Traditional testing methods, while often reliant on manual efforts supplemented by semi-automated tools, often fall short in addressing the dual challenges of identifying bugs effectively and minimizing the bug rate in generated code. This gap not only impedes the development process but also affects the overall software quality, leading to potential oversights and inadequate test coverage Ricca et al. 2024 ; Russo 2024 ; Feldt et al. 2018 . The industry’s shift towards continuous integration and deployment accentuates the need for more efficient testing processes. In this context, artificial intelligence (AI) emerges as a game-changer for enhancing software testing. Recent advancements in computational models, particularly large language models (LLMs) empowered by retrieval-augmented generation (RAG), offer new avenues for improving testing methods. These AI-driven technologies can analyze extensive codebases and detect complex patterns that may escape traditional testing approaches, thereby enabling the generation of precise and comprehensive test cases Alshahwan et al. 2023 . Given the progress of AI-assisted programming, there’s a significant opportunity to further apply these innovations to software testing and validation. By addressing the dual problems of increasing the bug detection rate and decreasing the bug rate in code generation, AI can play a pivotal role in transforming traditional testing paradigms. Figure 1: Architecture of Copilot for Testing : It proactively observes the local coding environment, retrieves code context, and generates context-aware prompts to interact with cloud-based LLMs and provide bug fix suggestions synchronized with codebase updates, aiming at more efficient and effective software testing with higher accuracy and coverage. We have previously launched and open-sourced Copilot for Xcode , 1 1 1 https://github.com/intitni/CopilotForXcode an AI-assisted programming tool for code auto-completion and Q&A, which was later re-licensed and assimilated into GitHub. 2 2 2 https://github.blog/changelog/2024-10-29-github-copilot-code-completion-in-xcode-is-now-available-in-public-preview/ Our initial system demonstrated the effectiveness of using contextual information to enhance the capabilities of LLMs in real-time code generation and problem-solving, which inspired us to extend the solution to software testing by transforming the challenge of generating code with fewer bugs into one of detecting and fixing bugs in the codebase. This paper introduces Copilot for Testing , an AI-assisted testing system that provides bug detection, fix suggestions, and automated test case generation directly within the development environment, in sync with codebase updates. Our proposed testing methodology integrates a context-based RAG mechanism that interacts dynamically with local coding environments and retrieves contextual information as an enhancement to the LLMs prompts. This interaction not only allows the system to adapt and refine testing strategies in real-time, responding to ongoing changes within the codebase, but also improves the performance of automated testing in efficiency, accuracy, and coverage. The main contributions of this work include: • We propose a software testing methodology powered by a context-based RAG module, achieving a 31.2% improvement in bug detection accuracy and a 12.6% increase in critical test coverage compared to the baseline. • We develop Copilot for Testing , an automated software testing system that seamlessly integrates with the development environment to deliver bug detection, fix suggestions, and test case generation in sync with codebase updates, achieving a 10.5% incremental acceptance rate for code suggestions in our user studies. 2 RELATED WORK 2.1 AI-Assisted Programming and Testing with Large Language Models The integration of AI, particularly through Large Language Models (LLMs), has revolutionized both programming and testing by automating and enhancing various tasks Beurer-Kellner et al. 2023 . AI-assisted programming tools like GitHub Copilot have demonstrated the potentials of LLMs in code completion and bug detection by understanding and generating code Chen et al. 2021 ; Nguyen and Nadi 2022 ; Wong and Tan 2024 . These dual problems—writing correct code and verifying code correctness—although distinct, share a common goal of reducing bugs and enhancing code quality. Our work extends this paradigm by offering a unified solution to both. 2.2 Automated Software Testing Automated software testing plays a crucial role in modern software development practices. Recent advancements include test case generation with machine learning techniques which reduces manual design efforts Guilherme and Vincenzi 2023 , automated test execution which can be integrated with continuous integration pipelines, and automated test results analysis which enhances decision-making processes regarding software quality Sumanth and Allam 2022 . Despite these advancements, challenges such as flaky tests and maintenance issues persist, highlighting the need for more robust testing methods and smarter maintenance strategies. Common metrics include accuracy, efficiency, and coverage Ricca et al. 2024 , which also guide our evaluation strategies. Figure 2: Overview of the context-based RAG module. The codebase is modeled as a graph, with individual nodes of code context embeddings. These embeddings are dynamically updated based on code changes initiated by the user and the outputs from LLMs. The toy example shows that when a change is made to node s_{2} , its embedding is updated, followed by updates to its neighboring nodes. The updated embeddings, which carry contextual information, are then utilized in prompt construction for the LLMs. 2.3 Search-Based Software Engineering Search-Based Software Engineering (SBSE) utilizes optimization algorithms to solve software engineering problems, where solutions are systematically searched based on a defined fitness function and problem representation Harman and Jones 2001 . It has been successfully applied to various software engineering tasks including test case generation, software refactoring, and requirement engineering. SBSE’s flexibility in defining fitness functions and representations makes it ideal for tailoring solutions to specific software engineering challenges, especially those involving large complex problem spaces. In our research, we model the problem within the SBSE framework and search for the optimal contextual information for the prompt construction. 2.4 Retrieval Augmented Generation Retrieval Augmented Generation (RAG) combines retrieval mechanisms with generative models to enhance output accuracy and contextual relevance Lewis et al. 2020 . Originally developed for natural language processing, RAG has shown promise in software engineering for tasks like bug detection Perera et al. 2020 . By integrating relevant code snippets or bug reports, RAG provides essential context to Large Language Models, improving the precision of generated solutions. Our approach leverages