“ My productivity is boosted, but … ” Demystifying Users’ Perception on AI Coding Assistants

Yunbo Lyu 2 , Zhou Yang 3 1 , Jieke Shi 2 , Jianming Chang 4 , Yue Liu 2 , David Lo 2 · 2025

Read “ My productivity is boosted, but … ” Demystifying Users’ Perception on AI Coding Assistants by Yunbo Lyu 2 , Zhou Yang 3 1 , Jieke Shi 2 , Jianming Chang 4 , Yue Liu 2 , David Lo 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-12285.txt.

Introduction

Yunbo Lyu and co-authors mine user reviews to learn what developers truly love and hate in AI coding assistants. Tools like GitHub Copilot now sit in most code shops, yet lab tests miss daily use. The team found 1,085 such assistants in the store of a big code platform. Near all of them are new: over 90 percent came out in the last two years.

From 32 assistants with rich reviews, the team read a sample by hand and built a map of user concerns. The mood of each note on each concern was tagged, for praise or gripe. Users cheer speed, but they ask for more: assistants that grasp the task, bend to their ways, and sip, not gulp, machine power. Five plain rules for makers close the paper, drawn from those asks.

Catalog record

CREATORYunbo Lyu 2 , Zhou Yang 3 1 , Jieke Shi 2 , Jianming Chang 4 , Yue Liu 2 , David Lo 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-12285

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

“ My productivity is boosted, but … ” Demystifying Users’ Perception on AI Coding Assistants Yunbo Lyu 2 , Zhou Yang 3 1 , Jieke Shi 2 , Jianming Chang 4 , Yue Liu 2 , David Lo 2 Abstract This paper aims to explore fundamental questions in the era when AI coding assistants like GitHub Copilot are widely adopted: what do developers truly value and criticize in AI coding assistants, and what does this reveal about their needs and expectations in real-world software development? Unlike previous studies that conduct observational research in controlled and simulated environments, we analyze extensive, first-hand user reviews of AI coding assistants, which capture developers’ authentic perspectives and experiences drawn directly from their actual day-to-day work contexts. We identify 1,085 AI coding assistants from the Visual Studio Code Marketplace. Although they only account for 1.64% of all extensions, we observe a surge in these assistants: over 90% of them are released within the past two years. We then manually analyze the user reviews sampled from 32 AI coding assistants that have sufficient installations and reviews to construct a comprehensive taxonomy of user concerns and feedback about these assistants. We manually annotate each review’s attitude when mentioning certain aspects of coding assistants, yielding nuanced insights into user satisfaction and dissatisfaction regarding specific features, concerns, and overall tool performance. Built on top of the findings—including how users demand not just intelligent suggestions but also context-aware, customizable, and resource-efficient interactions—we propose five practical implications and suggestions to guide the enhancement of AI coding assistants that satisfy user needs. Email: {yunbolyu, jiekeshi, liuyue, davidlo}@smu.edu.sg, [email protected], [email protected] † † aftertitle: I Introduction AI coding assistants, 1 1 1 In this paper, the terms AI coding assistants, in-IDE AI coding assistants, assistant, and AI extensions are used interchangeably, referring to AI-powered extensions in the IDE, such as GitHub Copilot. such as GitHub Copilot, have revolutionized software development practices [ 1 ] . These assistants leverage Large Language Models (LLMs) to provide code suggestions, ranging from token-level to block-level code completion, bug fixing, testing, and code explanation—covering nearly every aspect of software development [ 2 , 3 , 4 ] . Their ability to accelerate software development and reduce workloads has led to widespread adoption [ 5 , 6 ] . Although AI coding assistants have seen rapid growth and widespread adoption, recent studies indicate that developers may spend over 50% of their coding time verifying AI-generated suggestions, which can increase cognitive load and introduce additional validation effort [ 1 ] . This highlights the importance of understanding software developers’ perceptions and needs. Such understanding enables assistant developers to better align their tools with actual user requirements and shape the future of software development more effectively. Recent studies explore developers’ perceptions through interviews [ 7 , 8 , 9 ] , surveys [ 10 , 5 ] , and user studies [ 11 , 12 , 1 ] . However, these studies face limitations, such as small or restricted samples, high costs [ 13 ] , predefined questions, or a narrow focus on only a few AI coding assistants [ 14 ] . Meanwhile, online resources such as app reviews have proven effective in bridging the gap between developers and users, as they often involve large user bases and extensive feedback [ 15 , 16 , 17 ] . Yet, few studies have mined reviews for AI coding assistants [ 18 , 19 ] , and those that do face three notable limitations: (1) They focus exclusively on GitHub Copilot, overlooking the broader ecosystem of AI coding assistants and missing opportunities for cross-tool comparisons. (2) Existing taxonomy are built from sources like GitHub Issues and Stack Overflow, which highlight technical problems raised by advanced users [ 20 ] . This limits insight into subjective experiences and positive feedback, failing to capture what users value—not just what they find problematic. (3) Novice or non-technical users, who may lack the skills or motivation to create GitHub issues, are more likely to leave reviews on the Visual Studio Code Marketplace, where feedback is easier to submit. These overlooked voices are essential for understanding real-world perceptions of AI coding assistants [ 20 , 21 ] . An unprecedented opportunity exists to address this research gap: thousands of AI coding assistants are available in the Visual Studio Code (VS Code) marketplace, accompanied by extensive user feedback. These real-world reviews provide a rich source of information from developers all over the world, offering insights into their needs and expectations. As the most widely used IDE globally—adopted by 74% of developers according to Stack Overflow’s 2024 survey [ 22 ] —VS Code hosts a diverse range of AI coding assistants [ 23 ] . This rich dataset enables comprehensive analysis of user perceptions across multiple assistants, offering insights at scale and providing robust validation and extension of prior findings. We first collect 1,085 AI coding assistants identified among 66,053 extensions in the VS Code marketplace [ 24 ] through a combination of manual and automatic labeling. We employ GPT-4o to assist in the labeling process and validate its performance on a statistically significant sample, achieving both precision and recall above 96%. Our analysis reveals that over 90% of AI extensions were released in 2023 and 2024, highlighting a recent surge in their adoption. Additionally, we find that a small number of AI extensions dominate the market, with the top 10 installed extensions accounting for 86% of total installs. Moreover, we find that AI coding assistants receive more engagement, as AI extensions receive significantly more feedback and have higher average installs. The first goal of this paper is to construct a taxonomy of the aspects users discuss regarding AI coding assistants. Such a taxonomy helps developers understand what users care about at a high level and which specific characteristics of AI extensions they are particularly concerned with. It also supports improvements in development, maintenance, and testing by highlighting key focus areas. To ensure a systematic taxonomy construction process, we follow best practices for app review analysis [ 25 ] . We begin by sampling 361 user comments—statistically representative of the 5,908 reviews collected from 32 AI extensions. We do not consider extensions with insufficient installations or reviews. During the coding process, we adopt a bottom-up merging approach following prior work [ 26 ] . We first assign one or more labels to each comment, then merge identical labels, group them into subcategories, and finally organize them into top-level categories. Instead of open card sorting, we use a hybrid card sorting [ 27 ] , as the top-level categories are informed by established literature in the mature field of app review mining. In our pilot coding of 30 reviews, we initially generated 11 fragmented top-level categories. The predefined structure helps ensure conciseness, with pilot labels naturally aligning with the chosen dimensions. The final taxonomy includes 8 top-level categories (functionality, usability, dependability, supportability, system performance, general experience, comparison, and pricing), 16 subcategories, and 62 distinct labels. We analyze user feedback on AI coding assistants and identify six key findings. Users value accurate suggestions but often criticize redundant, incomplete, or buggy outputs. While many report productivity gains, novices are positive than experienced developers.

Read the complete text