CppPerf: An Automated Pipeline and Dataset for Performance-Improving C++ Commits

Tommy Ho, Khashayar Etemadi, Zhendong Su · 2026

Read CppPerf: An Automated Pipeline and Dataset for Performance-Improving C++ Commits by Tommy Ho, Khashayar Etemadi, Zhendong Su (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10890.txt.

Introduction

Tommy Ho, Khashayar Etemadi, and Zhendong Su build CppPerf, a pipeline and benchmark for automatically repairing C++ performance bugs. Realistic, runnable benchmarks for speed fixes have lagged behind, since existing C++ sets lean on contest code while real-world repair sets target Python and .NET.

CppPerf-Mine sifts open-source C++ repositories for patches that cut run time, filters them structurally, classifies them with a language model, and rebuilds each one inside a reproducible container. The resulting CppPerf-DB holds 347 hand-checked patches from 42 mature projects, and a leading repair agent fixes only about one in seven. Repair researchers get a hard, honest yardstick for repository-scale speed work.

Catalog record

CREATORTommy Ho, Khashayar Etemadi, Zhendong Su
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19280
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10890

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CppPerf: An Automated Pipeline and Dataset for Performance-Improving C++ Commits Tommy Ho, Khashayar Etemadi, Zhendong Su Abstract . Recent progress in automated repair of performance bugs demands realistic, executable benchmarks. However, existing C++ performance benchmarks are largely built from competitive programming submissions, and recent real-world benchmarks predominantly target Python and .NET. To fill this gap, we present CppPerf-Mine , a configurable pipeline that mines execution-time-improving patches from open-source C++ repositories on GitHub by combining structural commit filtering, an LLM-based commit classifier, and a containerized build & test stage that produces fully reproducible Docker images for each patch. Using CppPerf-Mine , we build CppPerf-DB , a benchmark comprising 347 manually verified patches from 42 mature C++ repositories, 39% of which are multi-file, enabling the evaluation of repository-level repair tools. In our preliminary study, OpenHands correctly fixes only 13.5% of the patches in CppPerf-DB , confirming that real-world C++ performance repair remains an open challenge. CppPerf-Mine and CppPerf-DB are open-source and publicly available at: https://doi.org/10.5281/zenodo.20097425 . In addition, a demonstration video is available at: https://www.youtube.com/watch?v=nixlupIgSdM . 1. Introduction After years of advancements in the field of automated program repair (APR), which has been mainly focused on fixing functional bugs ( 11 ; 10 ) , recent research has started to build new techniques for automated repair of performance bugs ( 5 ; 15 ; 3 ) . One of the challenges in this area is the scarcity of benchmarks of real-world performance bug-fixing patches. The benchmarks of performance patches that are most commonly used for evaluating APR tools contain online competition programs ( 14 ; 9 ; 2 ) , introductory-level programs ( 12 ) , and generated programs ( 8 ) . Recently, several benchmarks have been proposed that are collected from real-world projects in Python ( 6 ; 13 ; 16 ; 15 ) , C# ( 4 ) , Java ( 19 ) , and C++ ( 14 ) . However, they are typically collected from a small number of projects and provide no configurable tool for extending them according to user needs ( 13 ; 19 ) . In this paper, we introduce CppPerf-Mine , a novel and configurable tool for building benchmarks of executable performance bug-fixing patches in C++ repositories. CppPerf-Mine searches the commit history of GitHub C++ repositories, filters them based on user-provided configurations, employs an LLM-based technique to identify commits that improve execution time, checks that the project builds and passes tests, and outputs the executable execution-time-improving commits as Docker images that provide full reproducibility. With this approach, CppPerf-Mine can build extensible benchmarks by collecting additional and newer commits. Using CppPerf-Mine , we build CppPerf-DB , a benchmark with 347 real-world executable C++ patches that fix execution-time bugs and are collected from 42 different repositories. The patches in CppPerf-DB are collected from repositories that have between 306 and 28,718 stars, and 39% (136/347) of them modify multiple files, making CppPerf-DB diverse and suitable for evaluating repository-level patch generation. Our preliminary study shows that OpenHands , a state-of-the-art agentic tool, fixes only 13.5% of the bugs in CppPerf-DB , indicating the challenging nature of automated repair of real-world performance bugs. In summary, we make the following contributions: • CppPerf-Mine , a configurable tool for building benchmarks of executable C++ performance bug-fixing patches. • CppPerf-DB , a benchmark with 347 real-world C++ execution time bug-fixing patches collected from 42 repositories that span 6 years of development history. • An evaluation of the diversity of patches in CppPerf-DB and the precision of CppPerf-Mine , and a preliminary study using CppPerf-DB to assess the effectiveness of state-of-the-art agentic tools in fixing C++ performance bugs. 2. CppPerf-Mine : The Pipeline CppPerf-Mine takes user-defined settings as input and uses them to find commits that fix execution time bugs in C++ repositories. For each such commit, it produces a containerized environment that builds and runs tests on the corresponding patch. Figure 1 . An overview of the CppPerf-Mine workflow. Figure 1 shows an overview of the CppPerf-Mine workflow, which has four main components: (1) collecting repositories, (2) collecting commits, (3) LLM-based commit classification, and (4) containerized building and testing. All components consider user-defined settings to select and build containerized execution-time-improving patches that meet user expectations. 2.1. Collecting Repositories and Commits CppPerf-Mine starts by crawling GitHub to select repositories that meet five requirements. First, the repository should have at least the minimum number of stars specified by the user to ensure that it is a solid and popular project. Second, C++ should be the primary language of the repository. Third, the project should have a CMakeLists.txt file at its root and use CMake to build, allowing us to build and test selected projects with a unified technique. Fourth, the repository must define executable tests within its CMake configuration, which are used to check that the patch does not break the functionality of the program. Fifth, we run the tests on the latest version of the project to confirm that they build and pass successfully. After selecting the repositories that meet the requirements, CppPerf-Mine examines their commit histories and filters commits according to three structural criteria . First, the commit should be from the user-defined period of time. Second, the commit should not modify more than the maximum number of files specified by the user. Finally, the commit should only change C++ source code files; CppPerf-Mine excludes commits that change test files to ensure that a fixed set of tests can serve as a specification of program functionality in both the original and patched versions. Commits that meet these criteria are selected and passed on to the next step. 2.2. LLM-based Commit Classification CppPerf-Mine uses a two-phase LLM-based technique to assess whether a selected commit focuses on improving execution time. In the first phase, two LLMs are presented with the commit message and, if available, the description of the linked issue, and asked to determine if the commit is focused on execution time improvement by answering “Yes”, “No”, or “Maybe”. If both LLMs agree on “Yes” or “No”, we take that label as the final classification. This ensures that we classify a commit based on its commit message and linked issue description only when two LLMs are highly confident. If the LLMs do not agree, we proceed to the second phase, in which a single LLM is presented with the commit message, the issue description, and the code diff, and asked to respond with either “Yes” or “No”. The code diff provides detailed information about the modifications beyond the high-level commit message and issue description, but it can be large and costly to process. Therefore, we include it only for the limited number of commits that reach the second phase. 2.3. Containerized Building & Testing In the final step, CppPerf-Mine builds and runs the tests for each commit classified as execution-time-improving in a dockerized environment. It first fetches a base Docker image with an appropriate version of GCC and clones the original and patched versions of the repository into separate directories. Then it applies various heuristics to install the libraries required to build the project, falling back to an LLM-based iterative technique to detect and install missing libraries if the build fails. Finally, CppPerf-Mine

Read the complete text