Detecting Performance-Relevant Changes in Configurable Software Systems

Sebastian Böhm, Florian Sattler, Norbert Siegmund, Sven Apel · 2025

Read Detecting Performance-Relevant Changes in Configurable Software Systems by Sebastian Böhm, Florian Sattler, Norbert Siegmund, Sven Apel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17271.txt.

Introduction

Sebastian Böhm and three co-authors cut the cost of performance testing for configurable software. Systems with many options need testing across configurations, but remeasuring everything after each change is expensive, while sampling configurations can miss regressions that hit only a few of them. Their tool, ConfFLARE, takes a different route: it estimates whether a change can affect performance by tracing data flow into performance-relevant code, and identifies which features participate in those interactions.

Profiling then focuses on the configurations those features touch. On synthetic and real-world systems, ConfFLARE caught performance regressions in almost every case, found the relevant features nearly always, and cut the configurations to test by roughly three quarters or more. The savings amount to hours of testing time per change, without giving up completeness where it matters.

Catalog record

CREATORSebastian Böhm, Florian Sattler, Norbert Siegmund, Sven Apel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17271

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detecting Performance-Relevant Changes in Configurable Software Systems Sebastian Böhm, Florian Sattler, Norbert Siegmund, Sven Apel Abstract . Performance is a volatile property of a software system and, as the system evolves, its performance behavior may change, too. As a consequence, frequent performance profiling is required to keep the knowledge about a software system’s performance behavior up to date. Repeating all performance measurements after every revision is a cost-intensive task, though. This is even more challenging in the presence of configurability (e.g., by user-selectable software features), where one has to measure multiple configurations of a given software system to obtain a comprehensive picture. Configuration sampling is a common approach to control the measurement cost. However, sampling cannot guarantee completeness and might miss performance regressions, especially if they only affect a small portion of the configuration space. As an alternative to solve the cost reduction problem, we present ConfFLARE : ConfFLARE estimates whether a change potentially impacts performance by identifying data-flow interactions with performance-relevant code. It then extracts the configurable software features that participate in such interactions, as these features are highly likely to influence how the performance-relevant code is executed. Based on the identified features, we can select a subset of relevant configurations to focus performance profiling efforts on. To evaluate the effectiveness of ConfFLARE in a performance regression setting, we conduct a study on both synthetic and real-world software systems. In particular, we have address three questions: How effective is ConfFLARE in detecting performance-relevant changes ( ) and in identifying relevant features for performance regression analysis ( )? And how much performance measurement effort can we potentially save with this information ( )? We find that ConfFLARE correctly detects performance regressions in almost all cases and identifies the relevant features in all but two cases. With this information, we are able to reduce the number of configurations to be tested on average by 79 % 79\% for synthetic and by 70 % 70\% for real-world regression scenarios saving hours of performance testing time. 1. Introduction The performance behavior of a software system is crucial for its success. The challenge is that it is a volatile property that changes frequently as the system in question evolves. Software evolution can easily lead to performance regressions that negatively affect user experience or operating costs ( 24 ) . Performance regression testing aims at revealing performance issues early such that they can be fixed before deployment ( 18 ) . Since running performance tests for every revision is often a cost-intensive task, it is desirable to estimate whether and how incoming changes affect the system’s performance behavior. Such an estimation prior to actual testing becomes even more important in the presence of configurability: If a software system is configurable—and most systems are configurable—it is not sufficient to test only multiple revisions, but also multiple configurations need to be tested as well to obtain a complete picture. This causes an exponential blowup of the number of performance measurement runs. As a result, continuous performance testing on every revision is infeasible for many configurable software systems such that only a sample of all configurations are tested instead ( 41 ) . The problem is that the success of configuration sampling relies on the chance that regressing configurations are contained in the selected sample. As a consequence, regressions can easily be missed, especially if they affect only a small fraction of the overall configuration space ( 41 ) . Many sampling strategies have been devised that offer different trade-offs between sample size and coverage ( 25 ; 37 ) . These strategies typically only consider properties of the configuration space, that is, which configurable software features can be combined in which way, but they do not take advantage of the regression setting because they lack knowledge about which configurations are affected by a change. Studying performance bugs in highly configurable software systems, 23 suggest to use static analysis to determine which configurations are impacted by performance-relevant code changes so that performance testing efforts can be focused on impacted configurations. In this work, we present ConfFLARE , an approach that achieves exactly that: In a first step, we employ a data-flow analysis to determine whether a change to the code base of a configurable software system interacts with performance-relevant code in the system. Only if such an interaction exists, we consider the change for performance analysis. In that case, we localize in a second step the part of the configuration space that is potentially affected by that change so that fewer configurations need to be analyzed. As with sampling, ConfFLARE is a heuristic approach that trades completeness for scalability (a change may affect performance-relevant code even without data flow). However, it is a heuristic that is informed by the program’s semantics and, thus, can more precisely select relevant configurations than traditional configuration sampling strategies. Figure 1 . Example of a performance-relevant code change. Function compress contains performance-relevant code ( ). The callout points to a change that was made to Line 2 ( ). This change has an influence on the performance behavior of region since it changes how the loop counter is calculated. The example contains two features ( ) named StrongCompression and Verbosity , controlled by the parameters strong and verbosity , respectively. ConfFLARE reveals that only StrongCompression affects the example’s performance by influencing the value of iterations . For illustration, consider a typical example of a code change that introduces a performance regression in Figure 1 , which we use throughout the paper to explain how ConfFLARE works. Here, we briefly outline our idea on solving this problem. First, it is important to note that we focus here on performance regressions that are caused due to a change in a program’s execution. We do not consider performance regressions that are caused by other external factors or side effects (e.g., I/O, cache effects, etc.). The example models a simple configurable compression tool that undergoes a change. It consists of a function doStuff that takes a vector of data elements as well as some additional parameters and compresses the data vector in two steps: First, getIterations computes the number of iterations that is later used by the compression algorithm. Second, the data get compressed by function compress . This function mainly consists of a loop that is executed for the previously calculated number of iterations . As the program evolves, a new revision changes how variable iterations is computed ( ). To judge whether this could lead to a performance regression, we need to decide whether the changes of the new revision influence performance-relevant code, that is code that is central to the program’s performance behavior. ConfFLARE does so by determining whether a change interacts with performance-relevant code via a flow of data from the changed code to performance-relevant code. We call such data-flow interactions performance-relevant interactions . If we do not detect such interactions, we conclude (heuristically) that the change is likely not performance-relevant and, thus, no performance tests are run. By contrast, if ConfFLARE does find a performance-relevant change, it localizes the part of the configuration space that is potentially affected by that change. It does so by determining which configurable software features are involved

Read the complete text