µOpTime: Statically Reducing the Execution Time of Microbenchmark Suites Using Stability Metrics
Nils Japke, Martin Grambow, Christoph Laaber, David Bermbach · 2025
Read µOpTime: Statically Reducing the Execution Time of Microbenchmark Suites Using Stability Metrics by Nils Japke, Martin Grambow, Christoph Laaber, David Bermbach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12878.txt.
Introduction
Nils Japke and co-authors presented a tool in 2025 to cut the running time of tiny performance tests for software. Such tests, called microbenchmarks, catch slowdowns early, but suites can take hours because each test is repeated many times for accuracy.
Their tool uses the results of one full run to set the smallest repeat count for each test that still gives stable readings. It works from static settings, not live tuning. On fourteen open source projects in two languages, it cut the measuring phase by up to about ninety-five percent. The best stability metric varied by project, and warm up phases mattered for one language. It still caught real slowdowns.
Catalog record
| CREATOR | Nils Japke, Martin Grambow, Christoph Laaber, David Bermbach |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12878
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
µOpTime: Statically Reducing the Execution Time of Microbenchmark Suites Using Stability Metrics Nils Japke, Martin Grambow, Christoph Laaber, David Bermbach Abstract . Performance regressions have a tremendous impact on the quality of software. One way to catch regressions before they reach production is executing performance tests before deployment, e.g., using microbenchmarks, which measure performance at subroutine level. In projects with many microbenchmarks, this may take several hours due to repeated execution to get accurate results, disqualifying them from frequent use in CI/CD pipelines. We propose µOpTime, a static approach to reduce the execution time of microbenchmark suites by configuring the number of repetitions for each microbenchmark. Based on the results of a full, previous microbenchmark suite run, µOpTime determines the minimal number of (measurement) repetitions with statistical stability metrics that still lead to accurate results. We evaluate µOpTime with an experimental study on 14 open-source projects written in two programming languages and five stability metrics. Our results show that (i) µOpTime reduces the total suite execution time (measurement phase) by up to 95.83% (Go) and 94.17% (Java), (ii) the choice of stability metric depends on the project and programming language, (iii) microbenchmark warmup phases have to be considered for Java projects (potentially leading to higher reductions), and (iv) µOpTime can be used to reliably detect performance regressions in CI/CD pipelines. Journal: TOSEM CCS: General and reference Measurement CCS: General and reference Performance CCS: Software and its engineering Software performance CCS: Software and its engineering Software testing and debugging ACM Reference format: Nils Japke, Martin Grambow, Christoph Laaber, and David Bermbach. 2025. µOpTime: Statically Reducing the Execution Time of Microbenchmark Suites Using Stability Metrics. ACM Trans. Softw. Eng. Methodol. 34, 8, Article 231 (October 2025), 26 pages. https://doi.org/10.1145/3715322 1. Introduction As modern software systems grow in complexity and size, they are prone to experiencing performance regressions. To detect such regressions before deploying to production, benchmarking is an effective measurement-based solution. Microbenchmarks are the benchmarking technique of choice on source code level, similar to unit tests for functional testing, which can be integrated into CI/CD ( CI/CD ) pipelines to automatically ensure performance requirements, ideally, after every code change. Growing software projects often lead to growing microbenchmark suites, i.e., collections of individual microbenchmarks that are executed sequentially and repeatedly to evaluate a software’s performance. However, frequently executing large microbenchmark suites in CI/CD pipelines is unrealistic because they often run for several hours or even days ( Grambow et al., 2022 ; Grambow et al., 2021 ; Laaber et al., 2020a ) . One reason for these extensive execution times is the conservatively-configured number of repetitions for each microbenchmark, i.e., more repetitions generally lead to more reliable results. Current standard practice is to configure repetitions across different levels (e.g., within a VM and across different VMs), where each level potentially adds noise to performance measurements (e.g., variability inherent to the microbenchmark, variability of cloud platform performance). This leads to a high overall number of executions of each microbenchmark to guarantee sufficient result accuracy. At the same time, executing microbenchmarks after they have reached a stable result is dispensable, as it does not improve the ability to detect performance regressions, and, consequently, wastes time and computational resources. The state of practice, however, is using the same universal repetition configuration for all microbenchmarks, which provides an optimization opportunity to reduce execution time. Previous research has focussed on reducing the execution time of microbenchmarks by executing the microbenchmarks until a statistical threshold is reached ( Georges et al., 2007 ; He et al., 2019 ) , dynamically reducing microbenchmark warmup phases ( Laaber et al., 2020a ; Traini et al., 2024 ) , and selecting subsets of microbenchmarks ( De Oliveira et al., 2017 ; AlGhamdi et al., 2016 ; AlGhamdi et al., 2020 ; Chen et al., 2022 ; Grambow et al., 2022 ; Grambow et al., 2021 ) . However, none of these approaches distinguish between different repetition levels, only reduce the benchmark warmup phase and neglect the measurement phase, or do not execute potentially regression-exposing microbenchmarks at all. In this paper, we propose µOpTime, an offline analysis approach which applies statistical methods to assess a microbenchmark’s result stability and determine an improved execution configuration for each microbenchmark in a suite. Our approach is the first to shorten the overall execution time of individual microbenchmarks across different repetition levels. This leads to each microbenchmark having an individual execution time in each repetition level, as opposed to a universally set execution time in standard practice. In contrast to existing approaches, which dynamically stop a benchmark during the execution when measurements are stable ( He et al., 2019 ; AlGhamdi et al., 2016 ; AlGhamdi et al., 2020 ; Laaber et al., 2020a ; He et al., 2021 ; Traini et al., 2024 ) , µOpTime statically creates an optimal measurement configuration based on measurements from a full suite execution once, which is then used in consecutive benchmark suite runs. µOpTime is orthogonal to the existing approaches, and could potentially be combined, e.g., a performance engineer might first use µOpTime to reduce the execution time of all microbenchmarks, then during runtime reduce the set of microbenchmarks to be used ( Grambow et al., 2021 ) , and finally dynamically optimize warmup time ( Laaber et al., 2020a ; Traini et al., 2024 ) . Leveraging this, µOpTime (i) executes the full microbenchmark suite with a full execution configuration once, (ii) then determines the execution configuration with the shortest execution time, which leads to reliable results for each microbenchmark individually, by simulating smaller execution configurations and calculating for which configurations microbenchmarks are below a stability threshold, and (iii) thus collects a set of short-running configurations for subsequent code changes. To evaluate our approach, we address the following research questions: RQ1: To which extent does µOpTime reduce the execution time of full microbenchmark suites without affecting their result accuracy? We apply our approach, as well as two baseline approaches (i.e., minimum and random repetition levels), to three Go projects and eleven Java projects with 4,304 microbenchmarks in total. Some Java projects originate from two different research data sets, which is why we introduce the terms data set 1 for the Go projects, and data set 2 and data set 3 for the two Java data sets. Crucially, the benchmarking results in data set 2 do not use warmup iterations in their result data, so that we can study the influence later (cf. RQ3). We observe that, compared to the baseline approaches, µOpTime provides better result accuracy, and handles the trade-off between execution time and accuracy better. Using µOpTime, we can reduce the execution time by up to 95.83\% for Go (median of 76.8\% ), by up to 35.96\% for Java projects in data set 2 (median of 13.08\% ), and by up to 71.74\% for Java projects in data set 3 (median of 26.78\% ), where at least 80% of the microbenchmarks being within 3% of their original result with the full configuration. Some configurations of µOpTime lead to higher time reductions, but especially for Java, this leads to lower accuracy (details in Section 4.2 ). µOpTime shortened the duration from 57.5 to 2.32 minutes for zap ,