Copilot Arena: A Platform for Code LLM Evaluation in the Wild

Wayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, Ameet Talwalkar · 2025

Read Copilot Arena: A Platform for Code LLM Evaluation in the Wild by Wayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, Ameet Talwalkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09328.txt.

Introduction

Wayne Chi and co-authors introduce Copilot Arena, a platform for evaluating code-generating language models in real use, in a 2025 paper. Judging how well these models code in the wild is a challenging problem with no clear solution, because laboratory benchmarks miss real developer needs and habits. Their platform embeds model comparison directly inside a developer's working environment.

Users pick between paired model outputs while doing actual programming work, and a sampling strategy keeps response latency low for smooth use. The platform also adapts its prompting scheme to support code completion functionality inside the editor. Preference data gathered this way reflect practical usefulness, giving model builders feedback that static test suites cannot provide.

Catalog record

CREATORWayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, Ameet Talwalkar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09328

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Copilot Arena: A Platform for Code LLM Evaluation in the Wild Wayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, Ameet Talwalkar Abstract Evaluating in-the-wild coding capabilities of large language models (LLMs) is a challenging endeavor with no clear solution. We introduce Copilot Arena, a platform to collect user preferences for code generation through native integration into a developer’s working environment. Copilot Arena comprises a novel interface for comparing pairs of model outputs, a sampling strategy optimized to reduce latency, and a prompting scheme to enable code completion functionality. Copilot Arena has served over 4.5 million suggestions from 10 models and collected over 11k pairwise judgements. Our results highlight the importance of model evaluations in integrated settings. We find that model rankings from Copilot Arena differ from those of existing evaluations, which we attribute to the more realistic distribution of data and tasks contained in Copilot Arena. We also identify novel insights into human preferences on code such as an observed consistency in user preference across programming languages yet significant variation in preference due to task category. We open-source Copilot Arena and release data to enable human-centric evaluations and improve understanding of coding assistants. VSCode Download https://lmarena.ai/copilot Github Repo https://github.com/lm-sys/copilot-arena 1 Introduction Figure 1: Copilot Arena is a platform for conducting realistic evaluations of code LLMs, collecting human preferences of coding models with real users, real tasks, and in realistic environments, aimed at addressing the limitations of existing evaluations. Figure 2: We introduce Copilot Arena, a VSCode extension to collect human preferences of code directly in a developer’s IDE. Copilot Arena enables developers to use code completions from various models. The system comprises a) the interface in the user’s IDE which presents paired completions to users (left), b) a sampling strategy that picks model pairs to reduce latency (right, top), and c) a prompting scheme that allows diverse LLMs to perform code completions with high fidelity. Users can select between the top completion (green box) using tab or the bottom completion (blue box) using shift+tab . As model capabilities improve, large language models (LLMs) are increasingly integrated into user environments and workflows. For example, software developers code with AI in integrated developer environments (IDEs) [ 1 ] , doctors rely on notes generated through ambient listening [ 2 ] , and lawyers consider case evidence identified by electronic discovery systems [ 3 ] . Increasing deployment of models in productivity tools demands evaluation that more closely reflects real-world circumstances [ 4 , 5 , 6 ] . While newer benchmarks and live platforms incorporate human feedback to capture real-world usage, they almost exclusively focus on evaluating LLMs in chat conversations [ 7 , 8 , 9 , 10 ] . Model evaluation must move beyond chat-based interactions and into specialized user environments. In this work, we focus on evaluating LLM-based coding assistants. Despite the popularity of these tools—millions of developers use Github Copilot [ 11 ] —existing evaluations of the coding capabilities of new models exhibit multiple limitations (Figure 1 , bottom). Traditional ML benchmarks evaluate LLM capabilities by measuring how well a model can complete static, interview-style coding tasks [ 12 , 13 , 14 , 15 ] and lack real users . User studies recruit real users to evaluate the effectiveness of LLMs as coding assistants, but are often limited to simple programming tasks as opposed to real tasks [ 16 , 17 , 18 ] . Recent efforts to collect human feedback such as Chatbot Arena [ 9 ] are still removed from a realistic environment , resulting in users and data that deviate from typical software development processes. We introduce Copilot Arena to address these limitations (Figure 1 , top), and we describe our three main contributions below. We deploy Copilot Arena in-the-wild to collect human preferences on code. Copilot Arena is a Visual Studio Code extension, collecting preferences directly in a developer’s IDE within their actual workflow (Figure 2 ). Copilot Arena provides developers with code completions, akin to the type of support provided by Github Copilot [ 11 ] . Over the past 3 months, Copilot Arena has served over 4.5 million suggestions from 10 state-of-the-art LLMs, gathering 11604 votes from 1642 users. To collect user preferences, Copilot Arena presents a novel interface that shows users paired code completions from two different LLMs, which are determined based on a sampling strategy that aims to mitigate latency while preserving coverage across model comparisons. Additionally, we devise a prompting scheme that allows a diverse set of models to perform code completions with high fidelity. See Section 2 and Section 3 for details about system design and deployment respectively. We construct a leaderboard of user preferences and find notable differences from existing static benchmarks and human preference leaderboards. In general, we observe that smaller models seem to overperform in static benchmarks compared to our leaderboard, while performance among larger models is mixed (Section 4 ). We attribute these differences to the fact that Copilot Arena is exposed to users and tasks that differ drastically from code evaluations in the past. Our data spans 103 programming languages and 24 natural languages as well as a variety of real-world applications and code structures, while static benchmarks tend to focus on a specific programming and natural language and task (e.g. coding competition problems). Additionally, while all of Copilot Arena interactions contain code contexts and the majority involve infilling tasks, a much smaller fraction of Chatbot Arena’s coding tasks contain code context, with infilling tasks appearing even more rarely. We analyze our data in depth in Section 5.1 . We derive new insights into user preferences of code by analyzing Copilot Arena’s diverse and distinct data distribution. We compare user preferences across different stratifications of input data (e.g., common versus rare languages) and observe which affect observed preferences most (Section 5.2 ). For example, while user preferences stay relatively consistent across various programming languages, they differ drastically between different task categories (e.g. frontend/backend versus algorithm design). We also observe variations in user preference due to different features related to code structure (e.g., context length and completion patterns). We open-source Copilot Arena and release a curated subset of code contexts. Altogether, our results highlight the necessity of model evaluation in realistic and domain-specific settings. 2 System Design Copilot Arena is a VSCode extension that provides users with pairs of inline code completions from various LLMs. In return, users provide their votes on which completion is better suited for their task. To avoid interrupting user workflows, voting is designed to be seamless —users use keyboard shortcuts to quickly accept one of the two completions into their code, which we interpret as a vote in favor of the underlying model that produced it. Designed to allow for developer’s day-to-day usage, the three core components of Copilot Arena (Figure 2 ) are 1) the User Interface, 2) Model Sampling, and 3) Model Prompting. 2.1 User Interface Traditional code completion tools (e.g., GitHub Copilot [ 11 ] ) only show one completion at a time. However, showing two code completions simultaneously enables us to collect preference judgments on the same context [ 9 , 19 ]

Read the complete text