BenLOC: A Benchmark for Learning to Configure MIP Optimizers

Hongpei Li †,, Ziyan He †,, Yufei Wang †,, Wenting Tu ∗,, Shanwen Pu, Qi Deng, Dongdong Ge · 2025

Read BenLOC: A Benchmark for Learning to Configure MIP Optimizers by Hongpei Li †,, Ziyan He †,, Yufei Wang †,, Wenting Tu ∗,, Shanwen Pu, Qi Deng, Dongdong Ge (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02752.txt.

Introduction

A team of co-authors wrote a paper in two thousand twenty-five on setting up solvers for hard planning problems. Such solvers, for mixed-integer programming, have huge numbers of settings, and the right settings change from problem to problem. Claims about learned settings have often rested on uneven tests and leaked data.

Their benchmark, BenLOC, offers a shared, open pipeline for learning settings problem by problem. It fixes how data sets are picked, how training and test data are split, and which baselines count. Using it, the team compared classic learning models with hand-built features against deep learning methods on five known data sets.

Catalog record

CREATORHongpei Li †,, Ziyan He †,, Yufei Wang †,, Wenting Tu ∗,, Shanwen Pu, Qi Deng, Dongdong Ge
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02752

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BenLOC: A Benchmark for Learning to Configure MIP Optimizers Hongpei Li †,, Ziyan He †,, Yufei Wang †,, Wenting Tu ∗,, Shanwen Pu, Qi Deng, Dongdong Ge Abstract The automatic configuration of Mixed-Integer Programming (MIP) optimizers has become increasingly critical as the large number of configurations can significantly affect solver performance. Yet the lack of standardized evaluation frameworks has led to data leakage and over - optimistic claims, as prior studies often rely on homogeneous datasets and inconsistent experimental setups. To promote a fair evaluation process, we present BenLOC, a comprehensive benchmark and open-source toolkit, which not only offers an end-to-end pipeline for learning instance-wise MIP optimizer configurations, but also standardizes dataset selection, train-test splits, feature engineering and baseline choice for unbiased and comprehensive evaluations of the effectiveness. Leveraging this framework, we conduct an empirical analysis on five well-established MIP datasets and compare classical machine learning models with handcrafted features against state-of-the-art deep-learning techniques. The results demonstrate the importance of datasets, features and baseline criteria proposed by BenLOC and the effectiveness of BenLOC in providing unbiased and comprehensive evaluations. † † footnotetext: \dagger These authors contributed equally. † † footnotetext: * Corresponding author. † † footnotetext: \ddagger Email: [email protected] (Hongpei Li), [email protected] (Ziyan He), [email protected] (Yufei Wang), [email protected] (Wenting Tu), [email protected] (Shanwen Pu), [email protected] (Qi Deng), [email protected] (Dongdong Ge). 1 Introduction Mixed-Integer Programming (MIP) is a widely used mathematical optimization framework for solving real-world problems such as production scheduling, transportation planning, and revenue management [ 6 ] . However, the NP-hard nature of MIP poses significant challenges, even for advanced MIP optimizers like GUROBI, CPLEX, and SCIP, especially when addressing large-scale, real-time industrial problems asking for rapid response [ 3 ] . Therefore, improving the efficiency of MIP solvers remains a popular topic of operations research. Since MIP instances often share patterns and characteristics, machine learning (ML) approaches have emerged to enhance state-of-the-art solvers [ 20 , 10 , 15 ] . While many studies use GNNs to optimize branch-and-bound and cutting plane methods, integrating ML into core algorithms is challenging due to the complexity of modifying well-optimized solvers. Thus, recent research has increasingly focused on ML-based instance-wise configuration, which operates externally without altering the solver’s internal routines [ 18 ] . MIP solvers offer configurable settings for components like relaxations, heuristics, cutting planes, and branching strategies, all of which significantly impact solution quality and efficiency. Despite existing default settings and tuning tools, identifying optimal configurations for specific instances remains challenging and often requires extensive manual effort and expertise [ 17 ] . Although there are some works focusing on the automatic configuration of optimizers, the lack of a unified evaluation standard and benchmark has limited their effectiveness in the following ways: (1) coverage of datasets is not comprehensive, as some rely on homogeneous datasets that fail to represent real-world variability; (2) data splitting methods, such as permutation-based splits, introduce data leakage, thereby inflating performance estimates; and (3) the lack of a consistent, data-driven baseline raises difficulties in fair comparisons. With a thorough analysis of possible exaggerated performance results brought by existing methodologies, we propose a standard benchmark for ML-based optimizer automatic configuration by explicitly demonstrating: (1) a data-driven strategy for selecting datasets and baselines for unbiased comparisons; (2) the potential of underutilized dynamic handcrafted features, along with careful consideration of the extra time costs incurred when using features from solving process; also, we publish optimizer automatic configuration libraries and datasets built on our proposed benchmark, to promote future evaluations and comparisons in this field. We summarize our primary contributions as follows. First, we present the first comprehensive collection of datasets tailored for learning MIP optimizer configurations, along with well-defined selection criteria to ensure their representativeness and benchmarking suitability. Second, we establish a standardized experimental framework that enables objective and consistent evaluation, mitigating exaggeration and bias. Third, we conduct the most comprehensive benchmarking of competing methods to date , from both feature and algorithmic perspectives, providing a clear comparison of their respective strengths and limitations. Finally, we release an open-source toolset for optimizer configuration , which includes datasets, feature extractors, and machine learning methods ranging from traditional Machine Learning to Deep Learning techniques. 2 Literature Review Instance-Specific Algorithm Configuration (ISAC) [ 19 ] is a foundational approach using learning models to predict solver configurations, which clusters instances based on hand-crafted features (e.g., problem size, constraint types) using G-means clustering and applies a genetic algorithm (GGA) to find optimal configurations within each cluster. Hydra-MIP [ 28 ] extends ISAC by iteratively selecting multiple configurations, enhancing performance on diverse MIP instances. Unlike these black-box approaches, Iommazzo et al. 2020b formulate the configuration task as a mixed-integer nonlinear program (MINLP) using Support Vector Regression (SVR) to learn a performance map, achieving notable improvements in Hydro Unit Commitment (HUC) problems. Table 1: Literature Review. Dataset abbreviations: SC = Set Cover, CA = Combinational Auction, CFL = Capacitated Facility Location, IS = Independent Set, PAC = Packing, BIN = Binary Packing, PLA = Planning, MC = Maximum Cut, MRP = Multiple RGV Path, COR = CORLAT, MIK = MIK, IP = Item Placement, LB = Load Balancing, ANO = Anonymous, MIRP = Maritime IRP, CLU = CLUREG, CLUUR = CLUREGURCW, ISAC = ISAC (new), HUC = Hydro Unit Commitment, CMIP = Common MIP, MIP = MIPLIB, PP = Production Planning, GMI = General MIP. Liter. Method Dataset Baseline Feature Imp Default ML DL Homo. Hetero. Handcraft Graph [ 10 ] GNN SC CA, CFL IS ✗ default, FSB SVMRank LMART, Trees GNN -74%-36% [ 26 ] RL PAC, BIN PLA, MC ✗ Heuristics: Random MV, MNV,LE static [ 21 ] GNN SC, IS, MRP COR, MIK IP, LB, ANO, MIRP ✗ default Random SMAC3 FBAS GNN 0-50% [ 28 ] Tree CLU, CLUUR ISAC MIX default, PARAMILS, MIPzilla static dynamic graph 0-18% [ 18 ] SVR HUC ✗ default static [ 27 ] GNN IP, ANO ✗ default GNN [ 25 ] GNN CA, CFL, SC, IS CMIP, MIP default SMAC ISAC static GNN -63%-22.9% (MIPLIB) [ 15 ] MLP SC, KP, PLA GMI, PP ✗ Manually designed heuristics static dynamic 28%-52% [ 4 ] Rfr FICO MIP Always LC static dynamic 9-11% Recent studies also leverage graph-based representations to better capture structural information in MIP instances. Valentin et al. 2022 propose a GNN-MLP hybrid model that represents MIP problems as bipartite graphs, enabling end-to-end prediction of optimal SCIP configurations. Similarly, Song et al. 2023 introduce DGCAC, which uses random-walk-based graph embeddings and auto-encoders to cluster MIP instances and apply SMAC [ 16 ] for cluster-wise configuration, outperforming manual feature-based methods. For presolving configurations, Liu et al. 2024a develop a framework, named L2P, that combines neural networks with simulated annealing, using residual networks to capture presolver interactions, resulting in significant performance gains on industrial benchmarks. Table 1 summarizes recent advancements in learning-based methods for MIP configuration. Despite their potential, most studies have been tested on simple synthetic or homogeneous datasets, with limited success on more complex benchmarks like MIPLIB [ 21 ] . Moreover, performance comparisons often focus only on default solver settings, overlooking more rigorous comparisons against optimal configurations for specific instances or datasets. This lack of comprehensive evaluation raises concerns about the generalizability

Read the complete text