SelfRACG: Enabling LLMs to Self-Express and Retrieve for Code Generation
Qian Dong, Jia Chen, Qingyao Ai, Hongning Wang, Haitao Li, Yi Wu, Yao Hu, Yiqun Liu, Shaoping Ma · 2025
Read SelfRACG: Enabling LLMs to Self-Express and Retrieve for Code Generation by Qian Dong, Jia Chen, Qingyao Ai, Hongning Wang, Haitao Li, Yi Wu, Yao Hu, Yiqun Liu, Shaoping Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19033.txt.
Introduction
Qian Dong, Jia Chen, and seven colleagues let a language model ask for its own help when writing code, in a paper published in two thousand twenty-five. Retrieval-augmented code generation normally fetches code snippets that look like the text already written, then uses them to draft the next piece.
That match often fails, because the next fragment follows logically rather than textually, so its content drifts from what came before. An outside retriever cannot guess what the model actually needs next. In the team's design, SelfRACG, the model states its own information need in words. A retriever then hunts for code that meets that stated need. A two-stage training plan teaches models to express those needs well. The shift puts the model in charge of its own lookup. The authors aim for code assistants that pull in the right references at the right moment.
Catalog record
| CREATOR | Qian Dong, Jia Chen, Qingyao Ai, Hongning Wang, Haitao Li, Yi Wu, Yao Hu, Yiqun Liu, Shaoping Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19033
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SelfRACG: Enabling LLMs to Self-Express and Retrieve for Code Generation Qian Dong, Jia Chen, Qingyao Ai, Hongning Wang, Haitao Li, Yi Wu, Yao Hu, Yiqun Liu, Shaoping Ma Abstract Existing retrieval-augmented code generation (RACG) methods typically use an external retrieval module to fetch semantically similar code snippets used for generating subsequent fragments. However, even for consecutive code fragments, the content often diverges due to logical progression, resulting in a content gap. This gap undermines the performance of current RACG methods, as external retrieval modules based on content matching fail to infer the specific information need of LLMs to generate the next code fragment. Therefore, we propose SelfRACG , a novel paradigm that enables large language models (LLMs) to Self -express their information needs to enhance RACG . Specifically, SelfRACG includes an information need expression module and a two-stage information need-guided training strategy, which encourages LLMs to express their information need. Extensive experiments demonstrate that SelfRACG can retrieve external knowledge that better aligns with the LLM’s own information needs, resulting in superior generation performance compared to vanilla RACG. Moreover, both the training and deployment costs for retrieval in our framework are much lower than those of the strongest retrieval model. 1 1 1 https://github.com/CSQianDong/SelfRACG 1 Introduction Figure 1: Comparison of vanilla RACG vs. SelfRACG. Current retrieval-augmented code generation (RACG) techniques typically employ external retrieval modules, such as BM25 Robertson et al. 2009 or embedding-based retrieval models Muennighoff et al. 2024 ; Lee et al. 2024 , to identify code fragments that closely align in content with preceding codes, thereby enhancing the generation of subsequent fragments. However, even consecutive code fragments can differ significantly in content due to the progression of logic. As shown in Figure 1 , the preceding code fragment might implement a function for data pre-processing, such as cleaning or normalizing a dataset, while the subsequent fragment could define an entirely different function for training a machine learning model on the processed data. Although these two fragments are part of the same workflow, their content differs substantially, as each addresses a distinct functionality. Consequently, existing RACG methods struggle to effectively improve overall performance, as the external retrieval modules often fail to accurately fetch the knowledge required for code generation. These external retrievers typically rely on similarity to retrieve code fragments for RACG. However, the retrieved fragments are often functionally similar to the existing code, rather than specifically aligned with the LLM’s actual generation needs. Building on the observation that LLMs inherently possess the capability to generate the next token, we argue that the hidden states associated with the next token already encapsulate the information needs required for future content generation. Therefore, a critical research question is: Can LLMs express their own information needs for RACG? To enable LLMs to Self -express their information needs and perform RACG , our SelfRACG paradigm introduces two core components: an Information Need Expression (INE) module and a two-stage Information Need-Guided (ING) training strategy. Specifically, INE module is achieved by a parameter-efficient fine-tuning technique named Layer-wise Low-Rank Adaptation (L-LoRA). Through L-LoRA, we implement a retrieval-aware attention operation parallel to self-attention at each layer of the LLM. With minimal additional parameters and training costs, we extract the LLM’s information needs from the hidden state of the next token. Since the retrieval-aware attention operates completely in parallel with the LLM’s original self-attention, the next-token hidden states remain unaffected. Consequently, the original generation capabilities of the LLM can be fully preserved. The second key technique of SelfRACG is ING, a two-stage training strategy that enables LLMs accurately retrieve the information required for subsequent code generation. The first stage of ING leverages existing code from GitHub to create training samples, providing the LLM with foundational code retrieval skills. The second stage then uses a small amount of LLM-synthesized data to further train the model, aligning its retrieval capabilities with its own generation preferences. We conduct extensive experiments to validate our method on two comprehensive benchmarks using various top-tier code LLMs. The experimental results demonstrate that SelfRACG effectively bridges the content gap between retrieved and subsequent code fragments, thereby improving the quality of code generation. Moreover, both the training and deployment costs of our INE module for retrieval are much lower than those of the strongest retrieval module. The key contributions of our work can be summarized as follows: • We propose SelfRACG, a novel paradigm that enables LLMs to self-express their own information needs for RACG. • We introduce the INE module and ING training strategy to equip LLMs with the capability of expressing information needs at a low cost, without compromising the LLMs’ original capabilities. • We conduct comprehensive experiments on two benchmarks using multiple code LLMs, demonstrating effectiveness of our method. 2 Related Work 2.1 Code Large Language Models Recent advancements in code large language models (LLMs), such as Qwen2.5-Coder Hui et al. 2024 , OpenCoder Huang et al. 2024 , and DeepSeek-Coder Guo et al. 2024 , have demonstrated impressive capabilities in code generation. These models leverage vast amounts of code data to provide functional code snippets across diverse tasks. However, their performance often degrades when dealing with repository-level code completion, which requires fine-grained understanding of repository-specific contexts and dependencies Zhang et al. 2023 ; Wang et al. 2024 . To mitigate this issue, various retrieval-augmented generation methods have been proposed Zhang et al. 2023 ; Liu et al. 2024b ; Lu et al. 2022 . 2.2 Retrieval-Augmented Code Generation RACG has emerged as a promising paradigm for enhancing code generation by integrating external context into the generation pipeline Tan et al. 2024 ; Parvez et al. 2021 ; Gao et al. 2023 . Traditional RACG methods rely on retrievers like BM25 Robertson et al. 2009 or embedding-based models Muennighoff et al. 2024 ; Lee et al. 2024 to fetch contextually relevant code snippets from repositories. For example, RepoCoder Zhang et al. 2023 employs an iterative retrieval-generation framework to iteratively refine retrieved code snippets. Similarly, Repohyper Phan et al. 2024 introduces a semantic graph-based retrieval method to capture broader contextual relationships within repositories. RACG could address the limitations of parametric knowledge in LLMs. Despite the advantage, existing RACG approaches often struggle to bridge the content gap between retrieved codes and the model’s information needs, leading to suboptimal generation performance. 2.3 LLM-based Embedding Models Beyond generative tasks, LLMs have also been adapted for text embedding tasks, enabling them to serve as powerful embedding models. GritLM Muennighoff et al. 2024 unifies generative and embedding tasks within a single model through generative representational instruction tuning through extensive training costs. This approach enables GritLM to excel in both embedding and generation. Besides, gte-Qwen Li et al. 2023 , instructor Su et al. 2022 and NV-Embed Lee et al. 2024 extend embedding tasks with instruction tuning, enabling fine-grained control over embedding generation by explicitly encoding task-specific instructions. These works highlight the potential of LLMs in embedding tasks, presenting new possibilities for retrieval and semantic understanding. 3 Method In this section, we introduce our proposed SelfRACG framework. Section 3.1 introduces the Information Need Expression (INE) module, which enables LLMs to express their information needs from hidden states for retrieval. Section 3.2 outlines the two-stage Information Need-Guided (ING) training strategy, which is