Robust Dynamic Material Handling via Adaptive Constrained Evolutionary Reinforcement Learning

Chengpeng Hu, Ziming Wang, Bo Yuan, Jialin Liu, Chengqi Zhang, Xin Yao · 2025

Read Robust Dynamic Material Handling via Adaptive Constrained Evolutionary Reinforcement Learning by Chengpeng Hu, Ziming Wang, Bo Yuan, Jialin Liu, Chengqi Zhang, Xin Yao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16795.txt.

Introduction

Chengpeng Hu and co-authors train a learning system to assign moving jobs to vehicles in real time, in a paper from 2025. New transport tasks arrive on the fly. The goals are short total time and few late jobs, under hard delay limits.

Feedback only lands when all tasks are done, so the learner gets sparse reward. Their method keeps a flock of actors that explore in varied ways. Scores flag both bad rewards and rule breaks, which reins the policy in. Training time is handed out adaptively across past cases, so hard cases get their share. The result is a policy built to adapt when fresh tasks land mid-shift. Tests span many case sets.

Catalog record

CREATORChengpeng Hu, Ziming Wang, Bo Yuan, Jialin Liu, Chengqi Zhang, Xin Yao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE142 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-16795

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Dynamic Material Handling via Adaptive Constrained Evolutionary Reinforcement Learning Chengpeng Hu, Ziming Wang, Bo Yuan, Jialin Liu, Chengqi Zhang, Xin Yao Abstract Dynamic material handling (DMH) involves the assignment of dynamically arriving material transporting tasks to suitable vehicles in real time for minimising makespan and tardiness. In real-world scenarios, historical task records are usually available, which enables the training of a decision policy on multiple instances consisting of historical records. Recently, reinforcement learning has been applied to solve DMH. Due to the occurrence of dynamic events such as new tasks, adaptability is highly required. Solving DMH is challenging since constraints including task delay should be satisfied. A feedback is received only when all tasks are served, which leads to sparse reward. Besides, making the best use of limited computational resources and historical records for training a robust policy is crucial. The time allocated to different problem instances would highly impact the learning process. To tackle those challenges, this paper proposes a novel adaptive constrained evolutionary reinforcement learning (ACERL) approach, which maintains a population of actors for diverse exploration. ACERL accesses each actor for tackling sparse rewards and constraint violation to restrict the behaviour of the policy. Moreover, ACERL adaptively selects the most beneficial training instances for improving the policy. Extensive experiments on eight training and eight unseen test instances demonstrate the outstanding performance of ACERL compared with several state-of-the-art algorithms. Policies trained by ACERL can schedule the vehicles while fully satisfying the constraints. Additional experiments on 40 unseen noised instances show the robust performance of ACERL. Cross-validation further presents the overall effectiveness of ACREL. Besides, a rigorous ablation study highlights the coordination and benefits of each ingredient of ACERL. I Introduction In modern smart logistics such as flexible manufacturing systems and warehouse floors, the need of automated guided vehicles (AGVs) has grown fastly. Dynamic material handling (DMH) [ 1 , 2 ] involves scheduling a fleet of AGVs to serve dynamically arriving transporting tasks in real time. Transporting tasks typically include lifting and moving material between workstations using AGVs. The purpose of the DMH is to minimise makespan and tardiness, i.e., the maximal task finishing time and delay of tasks, in response to unexpected events and strict problem constraints. Due to the unexpected occurrence of dynamic events such as vehicle breakdowns and new tasks in DMH, the scheduling plans have to be determined frequently in real time [ 3 , 4 ] . Problem constraints such as the delay of tasks should be also guaranteed to ensure the operation of manufacturing [ 5 ] . Usually, some historical completed task records, e.g., previous task contexts and AGV information, are available in real life [ 6 , 7 ] . Training a decision policy using multiple instances consisting of historical records enhances the generalisation, but the trade-off across different instances is introduced. Selecting suitable instances at different training stages is crucial with consideration of limited computational budgets [ 8 ] . It is also hard to determine the unique contribution of each task assignment to the performance of the entire system since the overall performance can only be determined once all tasks are completed, which leads to a sparse feedback. Dispatching rule is a classic and common method for handling DMH [ 9 , 10 ] . Simple yet practicable mechanism makes them easy to deploy to real-world operations quickly. But this simplicity leads to limited performance and poor adaptability to real-world scenarios. Search-based methods, such as evolutionary algorithms (EAs) have been used for handling DMH when dynamic events occur, which restarts the search for a new solution [ 11 ] . However, a new search needs to be run from scratch for every single new scenario. The long search time merely meets the requirement of a fast response. Recently, reinforcement learning (RL) makes some promising progress in DMH [ 12 , 6 , 7 ] by providing prompt on-line responses with trained polices. In RL setting, DMH is formulated as a Markov decision process (MDP), where the reward function is manually constructed based on the makespan [ 6 ] . It takes massive effort and human knowledge to construct a “good” reward function since RL-based methods [ 12 , 7 ] often suffer from sparse feedback [ 13 ] . However, those designed reward functions cannot guarantee the consistence with the original objective function and fail to generalise on multiple instances. How to handle constraints such as the availability of AGVs and task delays in DMH is crucial. The work of [ 6 ] repeatedly sampled the task assignment until the sampled assignment is feasible. However, ensuring adherence to feasible task assignments is hard by re-sampling. It is also not plausible to satisfy the long-term task delay via simply penalising rewards [ 14 , 15 ] . To handle constraints, the work of [ 5 ] formulated the DMH problem as a constrained MDP [ 16 ] and proposed reward constrained policy optimisation with masking (RCPOM) approach. However, RCPOM suffers from sparse feedback from both reward and constraint sides. And the robust performance across multiple DMH problem instances is not fully addressed. A policy may perform well on the given instances, while failing on others, as it might not have been sufficiently trained on certain instances with limited computational budget [ 5 ] . In this paper, we consider DMH problem with uncertainties and sparse feedback in the context of multiple instances. We proposed a robust adaptive constrained evolutionary reinforcement learning (ACERL) approach to achieve real-time decision-making in DMH with adaptability and effectiveness. Unlike regular RL methods, ACREL inherits the gradient-free characteristic of natural evolution strategies (NES) [ 17 ] , which tackles sparse feedback in both rewards and constraint violation penalties intuitively. No gradient calculation related to the backpropagation is required. ACREL maintains a population of actors for diverse exploration. Intrinsic stochastic ranking using the rank-based fitness is proposed to evaluate the actors. Those rank-based fitness values facilitate the estimation of natural gradients, which implies the optimisation direction for maximising rewards and satisfying constraints simultaneously. The limited computational budget leads to a trade-off in the utilisation of multiple instances consisting of historical records for training a robust policy. To tackle the trade-off, we propose a novel training mode that adaptively selects a subset of training instances with which a population of actors interacts. The selected instances introduce a bias in estimating the natural gradients after being ranked by the intrinsic stochastic ranking, which contributes to a remarkable performance among multiple unseen instances. Main contributions of this paper are summarised as follows: • We propose ACERL to address DMH with uncertainties and sparse feedback. ACERL balances the reward maximisation and constraint satisfaction, even when the agent receives only limited feedback. Given multiple instances with different contexts, ACERL still provides robust scheduling solutions in real time. • We experimentally demonstrate the limitations of using a single instance or randomly selecting from multiple instances for training decision policies. Under this observation, we propose an adaptive training mode that deploys an adaptive instance sampler to break the trade-off of multiple instances. The limited computational resources are allocated by adaptively choosing the most beneficial instances for training a robust policy. • ACERL requires

Read the complete text