Repository Structure-Aware Training Makes SLMs Better Issue Resolver
Zexiong Ma, Yanzhen Zou, Bing Xie · 2024
Read Repository Structure-Aware Training Makes SLMs Better Issue Resolver by Zexiong Ma, Yanzhen Zou, Bing Xie (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19031.txt.
Introduction
Ma and four co-authors drill small tongue tools on code bank forms to fix faults in this twenty twenty-four paper. Big tools fix bank faults well but cost much and raise hush fears. Small tools lag on such tasks for want of bank form sense.
Inside, past faults and their fix pulls build two drill file kinds. One kind drills step-wise fault place finds in the bank. One kind drills patch writes from the found fault's near code. Tests on two known benches show small tools gain hard from the drill. Long bank reads also rose in the same test rounds.
Catalog record
| CREATOR | Zexiong Ma, Yanzhen Zou, Bing Xie |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19031
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Repository Structure-Aware Training Makes SLMs Better Issue Resolver Thanks: Corresponding authors. Zexiong Ma, Yanzhen Zou, Bing Xie Abstract Language models have been applied to various software development tasks, but the performance varies according to the scale of the models. Large Language Models (LLMs) outperform Small Language Models (SLMs) in complex tasks like repository-level issue resolving, but raise concerns about privacy and cost. In contrast, SLMs are more accessible but under-perform in complex tasks. In this paper, we introduce ReSAT (Repository Structure-Aware Training), construct training data based on a large number of issues and corresponding pull requests from open-source communities to enhance the model’s understanding of repository structure and issue resolving ability. We construct two types of training data: (1) localization training data , a multi-level progressive localization data to improve code understanding and localization capability; (2) code edit training data , which improves context-based code editing capability. The evaluation results on SWE-Bench-verified and RepoQA demonstrate that ReSAT effectively enhances SLMs’ issue-resolving and repository-level long-context understanding capabilities. 1 Introduction Language Models (LMs) have been applied to various software development tasks (such as code completion Cop (2022) , code generation Chen et al. (2021) ; Li et al. (2022) ; Ma et al. (2024) ; Jiang et al. (2024) , and program repair Jin et al. (2023) ), and many LM-based s have been integrated into real-world development processes. Language models can be categorized into two types based on their scales: (1) Large Language Models (LLMs) , with a parameter size of 100B or more, which are typically commercial, closed-source models cha (2022) ; Cla (2024) ; gpt (2023) . (2) Small Language Models (SLMs) , with a parameter size of 13B or less, which are usually open-source models Team (2024) ; Guo et al. (2024) ; Team et al. (2024) ; Touvron et al. (2023) ; Lozhkov et al. (2024) . LLMs have stronger abilities but are usually only accessible via API, raising concerns about privacy leakage. SLMs, on the other hand, can be deployed on consumer-grade GPUs Team et al. (2024) but have poorer performance in more realistic, repository-level software development tasks (i.e., issue resolving Jimenez et al. (2024) ). Due to the poor performance of SLMs on repository-level tasks Liu et al. (2023) , mainstream repository-level automatic programming assistants Cognition (2024) all employ LLMs. SWE-Bench Jimenez et al. (2024) is a benchmark to evaluate the ability of automatic programming assistants to resolve issues. Many works have employed agent-based or pipeline-based systems to utilize LMs for repository-level issue resolving, and all the top-performing works on the SWE-Bench leaderboard employ LLMs gpt (2023) ; Cla (2024) . Issue resolving requires language models to understand repository-level code and model long-range dependency. Given a lengthy code context and an issue, LMs are required to locate the code segment relevant to the issue, and generate code edits for the segments. Due to the limitations of model size and training data volume, neither the pre-training nor the instruction-tuning Luo et al. (2023) phases have endowed SLMs with the capability to perform such complex tasks effectively. There are rich structural information in open-source repositories, which is not utilized during training process of SLMs. Consequently, a pressing research question arises: Could we leverage the structural information in open-source repositories to enhance the repository-level understanding and issue-resolving capabilities of SLMs? (a) (b) Figure 1: Typical issue resolving frameworks, both of them first localize the code snippets related to the issue, then edit the snippets to resolve the issue. In this paper, we propose Re pository S tructure- A ware T raining ( ReSAT ), to enhance the repository-level code understanding and issue resolving capabilities of SLMs. We crawl popular open-source projects from GitHub and the resolved issues within these projects, using them to construct localization and code edit training data. (1) We construct localization training data to improve the repository-level code understanding and feature localization abilities of SLMs. Based on different levels of localization granularity, we create three types of localization data: file-level, function-level, and line-level localization. (2) We also construct code edit training data to improve the SLMs’ context-based code editing ability. We utilize ReSAT to fine-tune two SLMs: CodeQwen1.5-7B-Chat Team (2024) and Deepseek-Coder-6.7B-Instruct Guo et al. (2024) , and apply them to two issue resolving frameworks: Agentless Xia et al. (2024) and RAG-SWE Jimenez et al. (2024) . We utilize SWE-Bench-verified to evaluate the impact of ReSAT on the issue resolving capabilities of SLMs. Additionally, we assess the impact of ReSAT on the repository-level long-context understanding capabilities of SLMs through the RepoQA Liu et al. (2024) benchmark. We also conduct ablation studies to verify the effectiveness of the two parts of ReSAT. In summary, this paper makes the following main contributions: 1) we propose ReSAT, a novel repository structure-aware training data construction approach, to improve the repository-level code understanding and issue resolving capibility of SLMs; 2) we apply ReSAT on two open-source SLMs, the experiments on SWE-Bench-verified and RepoQA benchmarks indicate the effectiveness of ReSAT; 3) we conduct analysis on the unresolved issues of ReSAT-trained SLMs, providing guidance for further improving SLMs’ issue resolving capabilities. 2 Background In this section, we will provide a brief introduction to SWE-Bench, and introduce two existing issue resolving frameworks. 2.1 SWE-Bench SWE-Bench Jimenez et al. (2024) is a benchmark designed to test the capabilities of language models (LMs) in solving real-world software engineering problems. It consists of tasks derived from real GitHub issues and their corresponding pull requests across 12 popular python repositories. LMs or LM-based programming assistants are provided with an issue description and a repository, and expected to generate code edits to the repository that resolve the issue. Each issue is associated with test cases that can be executed in a Docker environment, and the evaluation is based on whether the edited repository can pass the test cases. SWE-Bench differs from traditional code generation benchmarks Chen et al. (2021) ; Peng et al. (2024) ; Austin et al. (2021) by focusing on realistic software engineering tasks that require repository-level code understanding, making edits across multiple locations, and applying long-context reasoning. SWE-Bench-verified OpenAI (2024) is an updated benchmark proposed to address the evaluation problems in SWE-Bench. The tasks in SWE-Bench are derived from real GitHub issues, which introduces various problems, such as overly specific unit tests, vague problem descriptions, and complex environment setup requirements. To address these problems, OpenAI collaborated with the SWE-Bench team to create SWE-Bench-verified, an updated and more reliable issue resolving benchmark. All issue resolving experiments in this paper are conducted on SWE-Bench-verified. 2.2 Issue Resolving Framework The intelligence level of GPT-4 makes it feasible to develop repository-level automatic programming assistants Wang et al. (2024) ; Bairi et al. (2024) . Since the release of GPT-4, many efforts have been made to build issue resolving frameworks. Some approaches design framework that align with the software development workflow. Issue resolving framework divide complex task into subtasks, sequentially invoking LMs to complete different subtasks in order to achieve repository-level programming. Jimenez et al. (2024) propose the RAG-SWE framework. As shown in Figure 1(a) , RAG-SWE first utilizes a retriever to retrieve the files most similar to the issue description, then uses the retrieved files and the issue description