How Do Agentic AI Systems Address Performance Optimizations? A BERTopic-Based Analysis of Pull Requests DOI: XXXXXXX.XXXXXXX Conference: MSR ’26: Proceedings of the 23rd International Conference on Mining Software Repositories; April 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/2018/06
Md Nahidul Islam Opu, Shahidul Islam, Muhammad Asaduzzaman, Shaiful Chowdhury · 2025
Read How Do Agentic AI Systems Address Performance Optimizations? A BERTopic-Based Analysis of Pull Requests DOI: XXXXXXX.XXXXXXX Conference: MSR ’26: Proceedings of the 23rd International Conference on Mining Software Repositories; April 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/2018/06 by Md Nahidul Islam Opu, Shahidul Islam, Muhammad Asaduzzaman, Shaiful Chowdhury (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24630.txt.
Introduction
Md Nahidul Islam Opu and three colleagues study how AI coding agents handle performance work. Agents increasingly write and revise real software. Correctness has been studied; performance behavior less so. The team mines agent-authored pull requests for performance changes. Language-model detection and topic modeling do the sorting. Fifty-two performance topics emerge, grouped into ten broad categories.
The optimizations span many layers of the software stack. Type of optimization matters for the outcome. Acceptance rates and review times both shift with it. Timing matters too. Agents do most performance work during development, not maintenance. The study grounds debates about agent quality in evidence. Reviewers and tool builders learn where agent optimizations land well and where they stall.
Catalog record
| CREATOR | Md Nahidul Islam Opu, Shahidul Islam, Muhammad Asaduzzaman, Shaiful Chowdhury |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-24630
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Do Agentic AI Systems Address Performance Optimizations? A BERTopic-Based Analysis of Pull Requests DOI: XXXXXXX.XXXXXXX Conference: MSR ’26: Proceedings of the 23rd International Conference on Mining Software Repositories; April 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/2018/06 Md Nahidul Islam Opu, Shahidul Islam, Muhammad Asaduzzaman, Shaiful Chowdhury Abstract . LLM-based software engineering is influencing modern software development. In addition to correctness, prior studies have also examined the performance of software artifacts generated by AI agents. However, it is unclear how exactly the agentic AI systems address performance concerns in practice. In this paper, we present an empirical study of performance-related pull requests generated by AI agents. Using LLM-assisted detection and BERTopic-based topic modeling, we identified 52 performance-related topics grouped into 10 higher-level categories. Our results show that AI agents apply performance optimizations across diverse layers of the software stack and that the type of optimization significantly affects pull request acceptance rates and review times. We also found that performance optimization by AI agents primarily occurs during the development phase, with less focus on the maintenance phase. Our findings provide empirical evidence that can support the evaluation and improvement of agentic AI systems with respect to their performance optimization behaviors and review outcomes. 1. Introduction LLM-based software engineering is rapidly reshaping development practices, with autonomous agents now implementing features, fixing bugs, and submitting pull requests (PRs) to software repositories in-the-wild ( 25 ) . Beyond functional correctness, the performance of AI-generated code is critical due to its impact on scalability, efficiency, and user experience ( 6 ; 30 ) . As a result, prior work has evaluated the performance of LLM-generated code against human-written code ( 6 ; 36 ; 37 ) , reporting mixed outcomes: AI can match or exceed human performance in some settings ( 9 ; 36 ; 30 ) , but shows notable inefficiencies in others ( 1 ; 12 ; 38 ) . However, existing studies largely rely on outcome-based evaluations conducted in controlled settings ( 1 ; 12 ) , providing limited insight into how agentic AI systems address performance concerns within real-world development workflows. In particular, it remains unclear which types of performance issues AI agents target and at which layers of the software stack these optimizations are applied. Moreover, little is known about the Software Development Life Cycle (SDLC) activities in which AI agents introduce performance optimizations, or whether different optimization types influence pull request acceptance rates and review times. Addressing these gaps is essential for enabling deeper empirical analyses that can inform the design and deployment of performance-aware LLM-based agents ( 30 ; 38 ) . Motivated by these gaps, we answer the following three research questions by analyzing a set of AI-generated performance-related pull requests as provided by the publicly available AIDev dataset ( 25 ) . RQ1: What categories of performance optimizations do agentic AIs address? Analyzing optimization categories reveals whether agents focus on localized changes or span the software stack, informing future model training and tooling. We identified 52 performance topics grouped into 10 categories, ranging from low-level optimizations to UI- and AI-related concerns, showing that agentic AIs operate across multiple stack layers. RQ2: How do PR rejection rates and review times vary across categories? Examining PR outcomes highlights areas where agentic AIs are more readily trusted versus those needing improvement. We observed an overall rejection rate of 36.5% in performance-related PRs, which is significantly higher than in non-performance PRs (22.7%). Also, rejection rates and review latency are largely driven by the type of performance concern, with agents performing worst on UI , AI , and Analytics PRs. RQ3: What SDLC activities are associated with AI-generated performance-related PRs? We found that performance-related PRs are more associated with feature implementation (i.e., development activities) and comparatively less associated with bug fixes, refactoring, and testing. This may indicate a potential gap in AI-assisted performance optimization during the maintenance phase. To enable replication, we share our data and code publicly. 1 1 1 https://github.com/SQMLab/LLM-performance Performance-related PRs grouped into 10 categories. Figure 1. Performance-related PRs grouped into 10 categories. Performance-related PRs grouped into 10 categories. 2. Methodology We used the AIDev-POP subset of version v3 of the AIDev dataset ( 25 ) , obtained from Zenodo ( 24 ) , which comprises 33,596 agentic pull requests. We focus only on this subset as it captures higher-quality, real-world projects and has the potential to avoid toy repositories ( 22 ) . Also, it provides the metadata required for RQ2 and RQ3, including PR acceptance status, review timing, and SDLC activities. Identifying Performance-Related PRs. AIDev dataset provides LLM-generated PR type annotations for each PR, but only 340 PRs are labeled as perf , substantially under-representing performance-related changes. We found that many PRs labeled as bug fixes or features also address performance concerns. For instance, the PR titled ’Fix Gantt performance by removing useMouse from every column’ 2 2 2 https://github.com/shadcnblocks/kibo/pull/166 is labeled as a fix despite explicitly targeting performance. To address this limitation, we aimed to identify all PRs that express performance concerns, even if they are associated with other activities, such as bug fixes. Following prior works ( 11 ; 29 ; 33 ; 40 ) , we initially applied keyword-based filtering using 23 performance-related terms (e.g., “performance, “optimize”, “latency”). Although this process yielded 10,894 PRs, manual inspection of 100 randomly sampled PRs by the first author—a graduate student with 2+ years of industry experience—shows a 92% false positives, making this approach unsuitable. We therefore adopted an LLM-based detection approach and reformulated the task as a binary classification problem (performance vs. non-performance). Using the open-source gpt-oss-20B model ( 2 ) in a zero-shot setting ( 23 ) , we classified PRs based on their intent. The prompt instructs the LLM to label a PR as performance-related only when performance is expressed as a concern. To guide the LLM, the prompt also included the set of performance-related keywords; the full prompt is provided in the replication package. This process identified 1,160 performance-related PRs across 447 repositories, generated by five different agents. Manual inspection of 200 randomly sampled PRs independently by first two authors, graduate students with 2+ and 10+ years of industry experience as software engineers, revealed a 7.5% false positive rate, defined as cases where at least one author labeled a PR as NonPerformance. Observed agreement was 97.5%, with high inter-rater reliability as measured by Cohen’s kappa (0.79) ( 8 ) and Gwet’s AC1 (0.97) ( 17 ) . However, since we applied BERTopic with Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN), which detects outliers ( 27 ) , we did not have to filter these cases, thus avoiding another step of laborious manual annotation. We also found that our new dataset missed 61 PRs from the 340 perf -labeled PRs in AIDev. To enlarge the dataset size, we added those 61 samples, yielding a final dataset of 1,221 performance PRs, substantially larger than the original subset. Topic Modeling with BERTopic. We applied the BERTopic ( 16 ) due to its superior performance over traditional methods ( 13 ; 14 ; 15 ) and identify topics in the filtered PRs. Combining the title and body of a PR, we generated embeddings using Qwen3-Embedding-8B