LLavaCode: Compressed Code Representations for Retrieval-Augmented Code Generation
Daria Cherniuk, Nikita Sukhorukov, Nikita Sushko, Daniil Gusak, Danil Sivtsov, Elena Tutubalina, Evgeny Frolov · 2025
Read LLavaCode: Compressed Code Representations for Retrieval-Augmented Code Generation by Daria Cherniuk, Nikita Sukhorukov, Nikita Sushko, Daniil Gusak, Danil Sivtsov, Elena Tutubalina, Evgeny Frolov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19644.txt.
Introduction
Daria Cherniuk and six co-authors present CoRoVA, a way to keep the benefits of retrieved context in code completion without paying its speed cost. Retrieval-augmented completion pulls in related code from the repository, but the long prompts slow the time to the first generated token, which hurts inside an editor. CoRoVA compresses the retrieved context into a few small vectors instead.
Only a small projector module needs training, and the compressed vectors stay meaningful to the code model. Prediction quality improved while the prompt grew by just a handful of tokens. On completion tasks the time to first token dropped by twenty to thirty-eight percent compared with uncompressed retrieval, a practical gain for interactive coding tools.
Catalog record
| CREATOR | Daria Cherniuk, Nikita Sukhorukov, Nikita Sushko, Daniil Gusak, Danil Sivtsov, Elena Tutubalina, Evgeny Frolov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19644
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLavaCode: Compressed Code Representations for Retrieval-Augmented Code Generation Daria Cherniuk, Nikita Sukhorukov, Nikita Sushko, Daniil Gusak, Danil Sivtsov, Elena Tutubalina, Evgeny Frolov Abstract Retrieval-augmented generation has emerged as one of the most effective approaches for code completion, particularly when context from a surrounding repository is essential. However, incorporating context significantly extends sequence length, leading to slower inference—a critical limitation for interactive settings such as IDEs. In this work, we introduce LlavaCode, a framework that compresses code into compact, semantically rich representations interpretable by code LLM, enhancing generation quality while reducing the retrieved context to only a few compressed single-token vectors. Using a small projector module we can significantly increase the EM and ES metrics of coding model with negligible latency increase. Our experiments demonstrate that compressed context enables 20-38% reduction in Time-to-First-Token (TTFT) on line completion tasks compared to full-RAG pipelines. 1 Introduction Recently, more and more IDEs started to feature code completion as one of the central tools. Code editors such as Windsurf 1 1 1 Windsurf homepage and Cursor 2 2 2 Cursor homepage started integrating large language models (LLMs) to provide single- and multiline prediction, which substantially improve developer productivity, but they also impose strict latency requirements: even small delays in time-to-first-token (TTFT) break the interactive coding experience and using this feature becomes frustrating. Additionally, retrieval-augmented generation (RAG) lewis2021retrievalaugmentedgenerationknowledgeintensivenlp has been widely adopted to improve both QA and completion quality, since it allows models to incorporate external context such as documentation, relevant snippets of code or function declarations into the prompt (Figure 1(a) ). However, the additional tokens from retrieval significantly increase prompt processing time and, consequently, TTFT, making vanilla RAG less practical for latency-critical settings like code completion. A promising solution is context compression via embedding projection. Originally introduced in multimodal models such as Flamingo alayrac2022flamingovisuallanguagemodel and LLaVA liu2023visualinstructiontuning , these methods use a separate visual encoder and a lightweight projection module to map input image embeddings into a small set of tokens for the language model. Subsequent works, such as xRAG cheng2024xragextremecontextcompression , extended this idea to textual retrieval, showing that compressed representations can match vanilla RAG performance while reducing inference cost. Despite this progress, no prior work has applied embedding projection to the code completion task, where the latency–quality trade-off is especially severe. Furthermore, existing training objectives (e.g., cross-entropy) are poorly aligned with developer-relevant code generation quality metrics such as Exact Match (EM) and Edit Similarity (ES), limiting the effectiveness of current approaches. Additionally, we can incorporate other code modalities, such as Abstract Syntax Trees (AST), into the retrieved embeddings to enrich the representations with syntactic information. In this work, we address both challenges. We propose a LLaVA-like projection mechanism for incorporating retrieved embeddings into model input. This projection mechanism is trained without unfreezing of LLM, combined with reinforcement learning that directly optimizes EM and ES. This enables efficient context integration for code completion without sacrificing prediction quality. Our contributions are the following: • To the best of our knowledge, our approach is the first to apply LLaVA-like embedding projection to code completion tasks without embedder or LLM finetuning, resulting in significantly higher EM and ES scores with negligible latency increase compared to base model, while maintaining 20-38% better latency compared to full RAG. • We’ve experimented with incorporating additional code modalities such as ASTs to investigate whether alternative representations of code can improve representation quality. • We’ve shown the limits of conventional LLM training with cross-entropy loss and proposed using Reinforcement Learning to directly train the projector for target EM and ES metrics, without unfreezing the retrieved context embedder or code-generating LLM. All the code and weights for projector modules will be available under permissive license. Code will be published in LlavaCode github repository 3 3 3 https://github.com/KamikaziZen/LlavaCode . 2 Related Work 2.1 Coding LLMs StarCoder li2023starcodersourceyou introduced a family of code generation models, including larger LLMs optimized for code-centric dialogue and smaller ones tailored for code completion. Trained on the permissively licensed The Stack dataset kocetkov2022stack3tbpermissively , these models achieved strong performance, surpassing most prior approaches on both code completion and instruction-following benchmarks. The Qwen-2.5-Coder series hui2024qwen25codertechnicalreport represented another significant advancement in code-focused LLMs. Trained on a proprietary mixture of data, the models were released in sizes ranging from 0.5B to 32B parameters and were designed to support text completion, code chat, and fill-in-the-middle tasks. In our work, we use these models as reader LLM baselines for RAG and evaluate our proposed system with Qwen-2.5-Coder-1.5B as backbone LLMs. 2.2 Context compression methods Despite decoder-only transformer optimizations such as KV-Caching pope2022efficientlyscalingtransformerinference and more efficient attention implementations like GQA ainslie2023gqatraininggeneralizedmultiquery , time per-token inference latency still scales linearly with context size. Since Retrieval Augmented Generation lewis2021retrievalaugmentedgenerationknowledgeintensivenlp retrieves information from the knowledge base and puts it into the context of language models, this increases the context size that needs to be processed and subsequently increases end-to-end latency. In the paper xRAG cheng2024xragextremecontextcompression the authors propose an approach, which is similar to multi-modal language models training: they push the embedding vector of the retrieved text from textual encoder through a lightweight projector layer to align it with the reader model. The resulting architecture is trained in a two-stage manner. In the first stage, both the encoder and LLM are frozen, while the projection layer is trained with cross-entropy loss on paraphrases of the same document. During the second stage, the projector is trained on a mix of tasks such as reading comprehension, open-domain QA and summarization, adding self-distillation from RAG teacher via KL term alongside with usual negative log-likelihood loss. Models trained in such way perform competitively with vanilla RAG systems, while being much more efficient and having lower TTFT due to the reduction in prompt length. Our approach is conceptually similar to xRAG method. By using a LLaVA-like projection from the encoder to the code completion model, we compress the retrieved context and maintain good generation quality, while lowering the TTFT. However, due to the specificity of our domain, we applied additional techniques to increase code-specific metrics and quality of predictions. Furthermore, we train only the projector with both the encoder and reader LLM frozen in a single stage manner. 2.3 Embedding models for code Code-search embeddings are commonly obtained by converting a decoder-only language model into embedding model by training them to produce last token embeddings for code search via contrastive learning. One such model is Qwen3-Embedding-0.6B zhang2025qwen3embeddingadvancingtext , which was converted from Qwen3-0.6B yang2025qwen3technicalreport model. Initialized from a powerful pretrained decoder-only model, Qwen3-Embedding-0.6B shows competitive scores on MTEB muennighoff2023mtebmassivetextembedding benchmarks among similarly sized embedding models. Additionally, some of the encoder models were trained not only on pure text and code data, but also on structured graphs, retrieved from code, such as Data Flow Graphs (DFG) and Abstract Syntax Trees (AST). Examples of such models are GraphCodeBERT guo2021graphcodebertpretrainingcoderepresentations and UniXcoder guo2022unixcoderunifiedcrossmodalpretraining models, which joined both code, text and graph data to improve representation quality for code-understanding and retrieval tasks. We have evaluated representative models as encoders in our architecture to investigate how different modalities of code effect the projection quality. 2.4 Reinforcement Learning in Language Modeling Training language models solely for next-token prediction optimizes perplexity but not other objectives such as lack of toxicity,