Context-Aware CodeLLM Eviction for AI-assisted Coding

Kishanthan Thangarajah 1 , Boyuan Chen 1 , Shi Chang 2 , Ahmed E. Hassan 3 · 2025

Read Context-Aware CodeLLM Eviction for AI-assisted Coding by Kishanthan Thangarajah 1 , Boyuan Chen 1 , Shi Chang 2 , Ahmed E. Hassan 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18796.txt.

Introduction

Thangarajah, Chen, Chang, and Hassan present a plan for which code model to drop from memory in self-hosted coding tools. Their paper is from two thousand twenty-five. Firms host their own code models for privacy and speed, but chip memory is tight. Dropping the wrong model forces a slow reload.

Their plan, CACE, weighs more than how long since a model was used. It looks at load time, how much delay a task can bear, likely output length, and past and near-future demand. In tests with mixed tasks, CACE cuts wait for the first token and for the full reply. It also drops models far less often than rival plans.

Catalog record

CREATORKishanthan Thangarajah 1 , Boyuan Chen 1 , Shi Chang 2 , Ahmed E. Hassan 3
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-18796

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Context-Aware CodeLLM Eviction for AI-assisted Coding Kishanthan Thangarajah 1 , Boyuan Chen 1 , Shi Chang 2 , Ahmed E. Hassan 3 Abstract AI-assisted coding tools powered by Code Large Language Models (CodeLLMs) are increasingly integrated into modern software development workflows. To address concerns around privacy, latency, and model customization, many enterprises opt to self-host these models. However, the diversity and growing number of CodeLLMs, coupled with limited accelerator memory, introduce practical challenges in model management and serving efficiency. This paper presents CACE , a novel context-aware model eviction strategy designed specifically to optimize self-hosted CodeLLM serving under resource constraints. Unlike traditional eviction strategies based solely on recency (e.g., Least Recently Used), CACE leverages multiple context-aware factors, including model load time, task-specific latency sensitivity, expected output length, and recent usage and future demand tracked through a sliding window. We evaluate CACE using realistic workloads that include both latency-sensitive code completion and throughput-intensive code reasoning tasks. Our experiments show that CACE reduces Time-to-First-Token (TTFT) and end-to-end (E2E) latency, while significantly lowering the number of model evictions compared to state-of-the-art systems. Ablation studies further demonstrate the importance of multi-factor eviction in balancing responsiveness and resource efficiency. This work contributes practical strategies for deploying scalable, low-latency AI coding assistants in real-world software engineering environments. I Introduction CodeLLMs are a distinct category of Large Language Models (LLMs) specifically designed for coding tasks. They are trained on a vast number of repositories of code artifacts and related documentation [ 1 , 2 ] . AI-assisted coding with CodeLLMs has significantly reshaped software development, boosting productivity by reducing both coding time and errors [ 3 , 4 , 5 , 6 , 7 , 8 ] . AI-assisted coding enables developers to concentrate on higher-level design and problem-solving tasks rather than boilerplate implementation. With the rapid release of new models, the number and variety of available CodeLLMs continue to grow. However, not all models perform equally across different coding scenarios. Some CodeLLMs excel at generating code, while others are more effective at understanding or reasoning through complex logic, especially when tailored to specific programming languages [ 9 , 10 , 11 , 12 , 13 , 14 ] . Likewise, performance can vary across application domains such as web development, mobile, blockchain, game, distributed systems, or networking, where certain CodeLLMs demonstrate domain-specific strengths [ 15 , 16 , 17 , 18 ] . As a result, developers now have the flexibility to choose from a wide range of models optimized for their particular use cases, making model selection increasingly critical to achieving high performance and relevance in AI-assisted coding workflows. To boost developer productivity, many organizations are integrating AI-assisted coding tools powered by CodeLLMs [ 19 , 6 , 20 , 4 ] . Enterprise software across diverse domains often rely on developers working with multiple programming languages [ 21 , 22 , 23 , 24 , 25 , 26 ] . For instance, a typical full-stack application may involve front-end developers using JavaScript or TypeScript, back-end developers writing services in Java or Python, DevOps engineers authoring infrastructure code in Go, and automation tasks scripted in Bash. This diversity is driven by the specialization of tools and languages for different layers of the software stack. Consequently, an effective CodeLLM-based assistant must support multiple languages and provide fast, high-quality responses across a broad spectrum of coding tasks and codebases. In addition, due to growing privacy and security concerns [ 27 , 28 , 29 ] , these organizations are increasingly opting to self-host CodeLLMs, including fine-tuned ones on proprietary data, rather than relying on external model APIs. While self-hosting specialized CodeLLMs enhances privacy control and allows customization to meet enterprise-specific requirements, it introduces a new challenge: managing limited in-house accelerator resources in environments where the total number of CodeLLMs needed in use far exceeds available resources. As developers expect near-instantaneous feedback, especially for latency-critical tasks such as code completion, reducing the cold-start latency of LLM requests becomes essential. One common approach for existing LLM serving systems is to enable model multiplexing. Model multiplexing refers to concurrently hosting and dynamically managing multiple models on shared accelerator resources. This approach optimizes resource usage by loading, unloading, and routing inference requests among various model instances according to real-time demands. Frameworks such as Ray Serve [ 30 ] , Ollama [ 31 ] , ModelMesh [ 32 ] , and other state-of-the-art systems [ 33 ] exemplify this approach by dynamically loading, unloading, and routing inference requests among multiple model instances within constrained accelerator memory. These systems aim to maximize resource utilization and throughput, typically employing simple heuristics, such as Least Recently Used (LRU), for model eviction decisions. However, without context-aware strategies, they often encounter unnecessary data movement and cold-start latencies that negatively impact developer experience in latency-sensitive tasks. This makes efficient model eviction strategies a key requirement to ensure the most relevant models are loaded dynamically, cold starts are minimized, and both latency and resource costs are optimized. In this paper, we introduce C ontext- A ware C odeLLM E viction ( CACE ), a dynamic and context-aware model eviction policy designed specifically for efficient and responsive CodeLLM serving in enterprise environments. Unlike traditional eviction methods, such as LRU, CACE integrates multiple context-aware factors, including model loading times, recent usage patterns, task-specific latency requirements, and predicted future developer requests. Through a comprehensive evaluation using realistic developer workloads comprising both latency-sensitive code completion tasks and throughput-intensive code reasoning tasks, we demonstrate that CACE significantly reduces Time-To-First-Token (TTFT) and end-to-end (E2E) latency, while minimizing unnecessary data movement (i.e., model reload). Additionally, we conduct extensive ablation studies to isolate and analyze the relative impact of each individual factor, providing clear insights into the effectiveness of our multi-factor eviction strategy. Specifically, we explore the following research questions (RQs): • RQ1: Can CACE reduce the overall number of model evictions compared to conventional eviction strategies in a self-hosted CodeLLM environment? • RQ2: How effectively does CACE reduce TTFT and E2E latency in a self-hosted CodeLLM environment? • RQ3: What is the individual impact of each context-aware factors within the CACE eviction policy? To ensure transparency and reproducibility, we intend to make our solution and all associated datasets publicly available [ 34 ] . The open-source release will allow reviewers and researchers to evaluate, replicate, and build upon our work. The remainder of this paper is structured as follows: Section II presents the background and motivation for efficient, latency-sensitive CodeLLM serving in enterprise environments. Section III highlights key operational challenges and industry trends. Section IV describes the detailed design of our proposed CACE approach, including the eviction policy and algorithmic factors. Section V outlines the experimental methodology, datasets, workload patterns, and evaluation metrics. Section VI discusses about insights from the results potential future research directions. Section VII focus on related work, positioning our contributions within the landscape of existing solutions, including model multiplexing and memory management strategies. Section VIII explores potential threats to validity and limitations of our approach. Finally, Section IX concludes the paper and outlines promising directions for future research. II Background and Motivation II-A Latency in AI-Assisted Coding AI-assisted coding powered by CodeLLMs significantly enhances developer productivity through intelligent suggestions for tasks such

Read the complete text