Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective
Mingyu Huang, Peili Mao, Ke Li · 2024
Read Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective by Mingyu Huang, Peili Mao, Ke Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16888.txt.
Introduction
Three researchers argue that the settings space of flexible software should be studied as a landscape, in a 2024 paper. Modern programs ship with many options, and how options map to speed stays a black box. Past studies treated each setting as a lone data point, which hides features such as local peaks. The team built an open tool that mines the setting space as a graph.
They ran the tool on eighty-six million tested settings from three real systems under many workloads. Six main findings emerged about the shape of these landscapes. Valleys, peaks, and paths through the space change how tuners should search and how models should be built. The authors say the landscape view gives a fuller picture than point-by-point study. Their tool is open for others to use on their own systems.
Catalog record
| CREATOR | Mingyu Huang, Peili Mao, Ke Li |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16888
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective Mingyu Huang, Peili Mao, Ke Li Abstract . 1 1 1 Accepted as a conference paper at ISSTA 2025 . Please note that this is the peer-reviewed version and not the final camera-ready one, as further modifications are expected. Modern software systems are often highly configurable to tailor varied requirements from diverse stakeholders. Understanding the mapping between configurations and the desired performance attributes plays a fundamental role in advancing the controllability and tuning of the underlying system, yet has long been a dark hole of knowledge due to its black-box nature. While there have been previous efforts in performance analysis for these systems, they analyze the configurations as isolated data points without considering their inherent spatial relationships. This renders them incapable of interrogating many important aspects of the configuration space like local optima. In this work, we advocate a novel perspective to rethink performance analysis—modeling the configuration space as a structured “landscape”. To support this proposition, we designed GraphFLA , an open-source, graph data mining empowered fitness landscape analysis (FLA) framework. By applying this framework to 86 86 M benchmarked configurations from 32 32 running workloads of 3 3 real-world systems, we arrived at 6 6 main findings, which together constitute a holistic picture of the landscape topography, with thorough discussions about their implications on both configuration tuning and performance modeling. 1. Introduction Modern software systems have become increasingly sophisticated and highly configurable, offering numerous tunable options—for example, the Linux kernel has over 15,000 options. By tuning these configuration options , stakeholders are empowered with the flexibility to tailor a system to meet specific functional requirements. Yet on the other hand, these configuration choices can also, intentionally or unintentionally, impact various non-functional performance attributes such as execution time and memory consumption of the system ( Xu et al., 2015 ; Siegmund et al., 2015 ) . As a result, stakeholders are often overwhelmed by the complexity of the system under tuning and may resort to default settings, which may in turn lead to sub-optimal system behaviors. For example, ( Han and Yu, 2016 ) shows that 59\% configuration-related issues can cause severe performance concerns in modern configurable software systems. Therefore, a thorough understanding of the intricate relationship between configurations and system performance is essential for effective system control and has been a long-standing goal in software engineering ( Siegmund et al., 2015 ; Ha and Zhang, 2019a ; Guo et al., 2013 ; Sarkar et al., 2015 ; Guo et al., 2018 ; Mühlbauer et al., 2023 ; Jamshidi et al., 2017a ; Zhang et al., 2015 ; Cheng et al., 2023 ; Dorn et al., 2020 ; Gong and Chen, 2024b ) . However, achieving this understanding in practice is far from trivial due to the black-box nature of the configuration-performance mapping. To shed light on these black boxes, empirical performance analyses have been used to analyze the distribution of system performance across large configuration spaces and different scenarios ( Mühlbauer et al., 2023 ; Jamshidi et al., 2017a ) . While performance distributions can provide useful information regarding the system—such as trends, variability, skewness, and kurtosis—they typically consider the performance values of sampled configurations as a set of isolated data points (see panel C of Figure 1 on page 1 ). Inherently, however, each performance value is associated with one or more system configurations and different configurations are spatially related to each other within the configuration space (panel B of Figure 1 on page 1 ). As a result, the associated performance values also exhibit spatial distributions across the configuration space, in addition to their one-dimensional numerical distributions. The lack of such spatial information in performance analysis leaves many important aspects of the software configuration problem unexplored. For example, local optima can present challenges for configuration tuning ( Chen and Li, 2023 ; Zhou et al., 2024 ) . However, they are not directly observable from performance distributions since this analysis requires examining the spatial adjacency between configurations. Consequently, current analytical methods are unable to interrogate questions like “are performance local optima prevalent in the configuration space of a given system?” Similarly, a large variance in overall performance distributions does not necessarily imply notable performance fluctuations in local neighborhoods. Without spatial information, performance distributions alone cannot reveal “whether the search landscape is smooth or rugged?” Further, practitioners are often interested in the top-performing configurations. Though prominent performance values can be identified from performance distributions, it is not clear “how are the corresponding configurations distributed in the configuration space?” These then lead to our central proposition: By taking into account the spatial information regarding the configuration space, we can potentially unlock many new insights into the configuration-performance mapping of configurable software systems that were previously inaccessible through traditional performance analyses. Figure 1. In software configuration space (panel B ), configurations are spatially related to each other, and so are their associated performance values. Traditional performance analysis (panel C ) only considers the distribution of performance values as isolated data points. Our approach (panel A ), instead, additionally incorporates the neighborhood relationships between configurations, which are used to construct a configuration landscape that reveals the spatial distribution of performance values across the configuration space. Bearing this in mind, we propose a new perspective for performance analysis of configurable software systems by jointly considering the spatial relationships between configurations and their associated performance values. To this end, we leveraged the fitness landscape metaphor, a fundamental concept in the evolutionary biology ( de Visser and Krug, 2014 ; Aguilar-Rodríguez et al., 2017 ; Vaishnav et al., 2022 ; Papkou et al., 2023 ) pioneered by Wright nearly a century ago ( Wright, 1932 ) . In the evolutionary context, “fitness” refers to the ability of a phenotype, expressed by a given genotype, to survive and reproduce in a given environment, and the “landscape” describes the distribution of all fitness values across the entire genotype space. The resulting fitness landscape contains, in principle, all the information regarding the genotype-fitness mapping (e.g., the number and locations of performance peaks). More generally, this metaphor has been widely adapted for understanding the variable-objective mapping of black-box systems across various domains like metaheuristics ( Huang and Li, 2023a ; Huang and Li, 2023b ; Tayarani-Najaran and Prügel-Bennett, 2014 ; Prügel-Bennett and Tayarani-Najaran, 2012 ) , physical chemistry ( Doye, 2002 ; Wang and Verkhivker, 2003 ) and machine learning (ML) ( Pushak and Hoos, 2022 ; Mohan et al., 2023 ; Huang and Li, 2023c ) . In the configurable software context, a fitness landscape can be conceived as a (hyper-)surface as formed by system performance values across a high-dimensional configuration space (panel A of Figure 1 on page 1 ). Each point in this landscape represents a configuration, with its elevation indicating the fitness and similar configurations are located close to each other. Using this metaphor: • The goal of configuration optimization is then analogous to moving uphill on this landscape toward the global peaks • The goal of performance modeling is equivalent to obtaining an accurate approximation of this surface given a few observations Therefore, knowledge regarding this landscape is