Reinforcement Learning-based Self-adaptive Differential Evolution through Automated Landscape Feature Learning
Hongshu Guo, Sijie Ma, Zechuan Huang, Yuzhi Hu, Zeyuan Ma, Xinglin Zhang, Yue-Jiao Gong · 2025
Read Reinforcement Learning-based Self-adaptive Differential Evolution through Automated Landscape Feature Learning by Hongshu Guo, Sijie Ma, Zechuan Huang, Yuzhi Hu, Zeyuan Ma, Xinglin Zhang, Yue-Jiao Gong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18061.txt.
Introduction
Differential evolution is a workhorse method for hard optimization, but tuning it for each new problem takes expert effort. Hongshu Guo and co-authors automate that tuning in a 2025 paper. Their method, RLDE-AFL, learns from the search itself: a neural attention module reads the current crowd of candidate solutions and their scores, and turns them into features describing the problem's landscape.
A reinforcement learning agent then uses those features to pick the method's settings on the fly, choosing from a wide menu of search operators. Training the feature learner and the control policy together beat both advanced hand-tuned methods and recent automated rivals on synthetic and realistic test problems. The team has released the code openly for others to build on.
Catalog record
| CREATOR | Hongshu Guo, Sijie Ma, Zechuan Huang, Yuzhi Hu, Zeyuan Ma, Xinglin Zhang, Yue-Jiao Gong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18061
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning-based Self-adaptive Differential Evolution through Automated Landscape Feature Learning Hongshu Guo, Sijie Ma, Zechuan Huang, Yuzhi Hu, Zeyuan Ma, Xinglin Zhang, Yue-Jiao Gong Abstract . Recently, Meta-Black-Box-Optimization (MetaBBO) methods significantly enhance the performance of traditional black-box optimizers through meta-learning flexible and generalizable meta-level policies that excel in dynamic algorithm configuration (DAC) tasks within the low-level optimization, reducing the expertise required to adapt optimizers for novel optimization tasks. Though promising, existing MetaBBO methods heavily rely on human-crafted feature extraction approach to secure learning effectiveness. To address this issue, this paper introduces a novel MetaBBO method that supports automated feature learning during the meta-learning process, termed as RLDE-AFL, which integrates a learnable feature extraction module into a reinforcement learning-based DE method to learn both the feature encoding and meta-level policy. Specifically, we design an attention-based neural network with mantissa-exponent based embedding to transform the solution populations and corresponding objective values during the low-level optimization into expressive landscape features. We further incorporate a comprehensive algorithm configuration space including diverse DE operators into a reinforcement learning-aided DAC paradigm to unleash the behavior diversity and performance of the proposed RLDE-AFL. Extensive benchmark results show that co-training the proposed feature learning module and DAC policy contributes to the superior optimization performance of RLDE-AFL to several advanced DE methods and recent MetaBBO baselines over both synthetic and realistic BBO scenarios. The source codes of RLDE-AFL are available at https://github.com/GMC-DRL/RLDE-AFL . Conference: The Genetic and Evolutionary Computation Conference; July 14–18, 2025; Woodstock, NY ISBN: 978-1-4503-RLDE-AFLX-X/18/06 CCS: Computing methodologies Bio-inspired approaches CCS: Computing methodologies Reinforcement learning CCS: Computing methodologies Markov decision processes 1. Introduction In the last few decades, Evolutionary Computation (EC) methods such as Genetic Algorithm (GA) ( Holland, 1992 ) , Particle Swarm Optimization (PSO) ( Kennedy and Eberhart, 1995 ) and Differential Evolution (DE) ( Storn and Price, 1997 ) have become more and more eye-catching in solving Black-Box Optimization (BBO) problems lacking accessible formulations or derivative information in both academia and industry ( Ma et al., 2024d ) . However, according to the “No-Free-Lunch” (NFL) theorem ( Wolpert et al., 1995 ) , no single algorithm or algorithm configuration (AC) can dominate on all problems. Therefore, to enhance the optimization performance on diverse problems, researchers have manually designed various adaptive operator selection and parameter control methods, which achieve superior performance on BBO benchmarks ( Hansen et al., 2009 ; Mohamed et al., 2021 ; Ma et al., 2023 ) . However, the design of the operator and parameter adaptive mechanisms requires substantial experience and deep expertise in optimization problems and algorithms. One may need to adjust the configurations iteratively according to the problem characteristic and optimization feedback, consuming significant time and computational resources. To relieve the human effort burden, recent researchers introduce Meta-Black-Box Optimization (MetaBBO) which leverages a meta-level policy to replace the human-crafted algorithm designs in Algorithm Selection ( Guo et al., 2024 ) , Algorithm Configuration ( Ma et al., 2024b ; Lian et al., 2024 ; Sun et al., 2021 ; Sharma et al., 2019 ) , Solution Manipulation ( Li et al., 2024a ; Li et al., 2024c ; Wu et al., 2023 ; Li et al., 2024b ) and Algorithm Generation ( Chen et al., 2024 ; Ma et al., 2024c ; Zhao et al., 2024 ) . MetaBBO methods typically involve a bi-level architecture. In the meta level, given the optimization state, a neural network based policy determines an appropriate algorithm design for the low-level algorithm at each optimization generation. The resulting changes in objective values after algorithm optimization are returned to the meta-level policy as meta performance signal, which is then used to refine the meta policy ( Yang et al., 2024 ; Ma et al., 2024d ) . Given the meta policy \pi_{\theta} parameterized by \theta with the algorithm A on a set of problem instances \mathcal{I} , the objective for algorithm configuration is formulated as: (1) \mathbb{J}(\theta)=\mathop{\arg\max}\limits_{\theta\in\Theta}\mathop{\mathbb{E}}\limits_{f\in\mathcal{I}}\big[\sum_{t=1}^{T}Perf(A,\omega_{t},f)\big] where T is the optimization horizon, Perf(\cdot) is a performance metric function, \omega_{t}=\pi_{\theta}(s_{t}) is the algorithm design outputted by the policy and s_{t} is the optimization state at generation t . For algorithm configuration, \omega may correspond to operator selection, parameter values, or both. To maximize the expected performance on the problem distribution, machine learning methods such as Reinforcement Learning (RL) are widely adopted for policy training. Figure 1. The overview of the bi-level structure in RLDE-AFL. Though promising, these works still retain significant potential to further reduce the expertise burden and enhance the performance. The first limitation is the expertise dependent optimization state design, which requires substantial domain to develop informing and representative features for comprehensive configurations. Besides, while traditional BBO community have developed diverse operator variants, existing MetaBBO methods typically adopt only a small subset of these operators, leading to a limited strategy diversity. Moreover, the configuration spaces in existing MetaBBO methods are restricted, most works focus on exclusively operator selection ( Sharma et al., 2019 ; Tan and Li, 2021 ) or parameter control ( Sun et al., 2021 ; Ma et al., 2024b ) , failing to fully unleash the behavior diversity of the meta policies. To address these issues, we propose RLDE-AFL. Firstly, for the expertise-dependent state representation, we employ NeurELA ( Ma et al., 2024a ) , as shown in the top left of Figure 1 , a self-attention neural network based optimization information extractor to automatically analyze the problem landscape from the population and evaluation values, eliminating the need for manual feature design. The two-stage self-attention between the dimensions and individuals achieves generalizable and comprehensive state extraction. By integrating the mantissa-exponent based evaluation value representation, we further enhance the generalization ability of RLDE-AFL. Then, to further enhance the performance of RLDE-AFL, we integrate 14 DE mutation operators and 3 DE crossover operators with diverse optimization behaviours to form the candidate operator pool (illustrated in the bottom right of Figure 1 ). This diversity enables the agent to adaptively deploy distinct optimization strategies tailored to different problem instances. By incorporating parameter control, RL agent gains full control over the DE framework through optimization states derived from the automatic feature extractor, thereby unleashing the behavior diversity of RL agent to achieve superior performance. Finally, we conduct extensive experiments to demonstrate the effectiveness of RLDE-AFL on MetaBox ( Ma et al., 2023 ) benchmark problems. The comparisons with advanced traditional DE algorithms and RL-based DE methods confirm the superior performance of RLDE-AFL. The zero-shot performance on different dimensional problems, expensive problems and out-of-distribution realistic problems shows the robust generalization capabilities of RLDE-AFL, outperforming existing advanced traditional and learning based methods. The rest of this paper is organized as follows: Section 2 reviews the related works on existing traditional adaptive DE. Section 3 introduces the preliminary concepts on RL and NeurELA. In Section 4 , we present our technical details, including the Markov Decision Process (MDP) definition, network design and training process. The experimental results are presented in Section 5 , followed by a conclusion in Section 6 . 2. Related Works 2.1. Traditional AC for DE For DE algorithms, the algorithm configurations usually focus on selecting mutation and crossover operators and controlling the parameters of these operators. In vanilla DE ( Storn and Price,