MFTune: An Efficient Multi-fidelity Framework for Spark SQL Configuration Tuning
Beicheng Xu †∗ , Lingching Tung †∗ , Yuchen Wang † , Yupeng Lu † , Bin Cui † · 2026
Read MFTune: An Efficient Multi-fidelity Framework for Spark SQL Configuration Tuning by Beicheng Xu †∗ , Lingching Tung †∗ , Yuchen Wang † , Yupeng Lu † , Bin Cui † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16450.txt.
Introduction
Xu and co-authors tune Spark SQL settings without paying for full test runs. Spark's huge option space makes each full check costly. Cheap proxies like shrinking data break the link to real bottlenecks. Their MFTune instead splits fidelity by query subsets. Chosen subsets stand in for the whole workload at low cost. Density-based steps compress the knob space before search. Tests on TPC-H and TPC-DS beat five rival tuners.
Catalog record
| CREATOR | Beicheng Xu †∗ , Lingching Tung †∗ , Yuchen Wang † , Yupeng Lu † , Bin Cui † |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71935 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16450
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MFTune: An Efficient Multi-fidelity Framework for Spark SQL Configuration Tuning Thanks: * Beicheng Xu and Lingching Tung contribute equally to this paper. Beicheng Xu †∗ , Lingching Tung †∗ , Yuchen Wang † , Yupeng Lu † , Bin Cui † Abstract . Apache Spark SQL is a cornerstone of modern big data analytics. However, optimizing Spark SQL performance is challenging due to its vast configuration space and the prohibitive cost of evaluating massive workloads. Existing tuning methods predominantly rely on full-fidelity evaluations, which are extremely time-consuming, often leading to suboptimal performance within practical budgets. While multi-fidelity optimization offers a potential solution, directly applying standard techniques—such as data volume reduction or early stopping—proves ineffective for Spark SQL as they fail to preserve performance correlations or represent true system bottlenecks. To address these challenges, we propose MFTune, an efficient multi-fidelity framework that introduces a query-based fidelity partitioning strategy, utilizing representative SQL subsets to provide accurate, low-cost proxies. To navigate the huge search space, MFTune incorporates a density-based optimization mechanism for automated knob and range compression, alongside an adapted transfer learning approach and a two-phase warm start to further accelerate the tuning process. Experimental results on TPC-H and TPC-DS benchmarks demonstrate that MFTune significantly outperforms five state-of-the-art tuning methods, identifying superior configurations within practical time constraints. † † authors: . PVLDB Reference Format: PVLDB, 14(1): XXX-XXX, 2020. doi:XX.XX/XXX.XX † † This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected] . Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 14, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/Elubrazione/MFTune . 1. introduction (a) Performance distribution. (b) Fidelity correlation. Figure 1 . Analysis of multi-fidelity mechanism on TPC-DS (600GB). (a) Performance distribution of configurations evaluated within a 48-hour budget, comparing MFTune against a single-fidelity variant. (b) Fidelity correlation. We evaluate the representativeness of three proxy methods by randomly sampling 50 configurations from the search space. The proxies include: 1) Data Volume: scaling the dataset across {30,100,200,400,600} GB; 2) SQL Early Stop: executing the first {1/27,1/9,1/3,2/3,1} of the total SQLs; and 3) SQL Selection: selecting representative SQL subsets at fidelity levels 1/27 to 1 based on Section 6.1 . Each point represents the Kendall’s Tau correlation between proxy and full-fidelity performance, plotted against the average latency ratio (proxy latency relative to full-fidelity latency) across 50 configurations. The recent explosion of data has catalyzed the widespread deployment of large-scale query systems, such as Hive ( 47 ) , Presto ( 41 ) , and Spark SQL ( 2 ) . Among these, Spark SQL has emerged as a premier choice for industry-scale data processing due to its robust integration with the Spark ecosystem ( 60 ) and its superior performance in handling complex analytical workloads. It serves as a foundational component for critical downstream applications like database management ( 57 ) . However, the efficiency of Spark SQL is heavily dependent on more than 200 configuration knobs that govern system behaviors like memory management and shuffle partitioning ( 45 ) . For example, the knob spark.executor.memory specifies the amount of memory for an executor process. An excessively large value increases garbage collection time, whereas an excessively small value may cause out-of-memory errors. Because the default settings are often suboptimal for specific workloads and the search space is prohibitively large, manual tuning by database administrators is both time-consuming and far from optimal. To find optimal configurations for Spark SQL workloads, various automatic configuration tuning methods have been proposed ( 56 ; 42 ; 29 ; 26 ; 5 ; 8 ) . These methods typically adopt the Bayesian Optimization (BO) framework ( 44 ; 12 ) to sample promising configurations, which iteratively samples configurations and replay the workload to model the relationship between system performance and parameters. However, these methods suffer from several fundamental challenges when applied to Spark SQL: C1. Prohibitive evaluation costs. Spark SQL workloads typically handle massive data volumes and consist of multiple queries, leading to execution times that span hours or even days. However, existing tuning methods often rely on full-fidelity evaluations (executing the entire original workload) ( 8 ; 26 ; 29 ) , which limits the number of configurations that can be explored within a finite tuning budget. A potential solution to mitigate this bottleneck is to adopt Multi-Fidelity Optimization (MFO) ( 31 ; 27 ; 7 ) . MFO leverages low-fidelity evaluations, which are faster but approximate the full-fidelity results, to efficiently filter out poor configurations and guide the search toward optimal regions. Our preliminary analysis in Figure 1(a) demonstrates the impact of this mechanism: within a 48-hour budget, MFTune successfully evaluated 43 configurations with MFO. In sharp contrast, a single-fidelity variant (MFTune w/o multi-fidelity) evaluated only 29 configurations in the same time frame, with the majority falling into higher latency buckets or even resulting in execution errors. These illustrate that relying solely on full-fidelity evaluations makes it nearly impossible to explore the vast search space effectively within practical time constraints. C2. Inapplicability of directly applying existing MFO methods. The core challenge of MFO lies in ensuring that the low-fidelity proxies preserve the relative ranking of configurations observed under full fidelity. However, existing standard fidelity-partitioning techniques used in Automated Machine Learning (AutoML), such as training on a subset of data ( 31 ; 20 ; 30 ) or for fewer epochs ( 16 ; 30 ; 7 ) , do not transfer effectively to Spark SQL. As illustrated in Figure 1(b) , existing proxies such as Data Volume and Early Stop fail to maintain a reliable correlation with full-fidelity performance in Spark SQL. Reducing data volume leads to a sharp decline in correlation because performance does not scale linearly; smaller datasets often underutilize cluster resources and fail to trigger the true bottlenecks encountered at full scale. Similarly, SQL Early Stop—executing only the initial queries of a workload—fails to capture the diverse data coverage and operation types representative of the entire workload. In contrast, our proposed SQL Selection maintains a high Kendall-tau correlation (>0.8) even at extremely low fidelity levels. These findings demonstrate that customizing fidelity partitioning for the Spark SQL domain is essential, as generic AutoML proxies cannot maintain the ranking consistency required for effective tuning. C3. Huge search space and ineffective compression. With over 200 parameters, the search space of Spark SQL is prohibitively large for exhaustive exploration within a limited budget. Most existing works focus solely on reducing the number of knobs via sensitivity analysis or projection ( 56 ; 8 ; 54 ; 29 ) . Only a few methods ( 65 ; 46 ) have attempted to compress the value range. However, these approaches rely on simple bucketization or discrete historical boundaries, leaving them highly susceptible to outliers and unable to capture promising regions that were insufficiently explored in previous tasks. Consequently, achieving robust and fine-grained search space compression remains an open challenge. To address these challenges, we propose MFTune, an efficient multi-fidelity framework tailored for Spark