FM-EAC: Feature Model-based Enhanced Actor-Critic for Multi-Task Control in Dynamic Environments
Quanxi Zhou, Wencan Mao, Manabu Tsukada, John C.S. Lui, Yusheng Ji · 2025
Read FM-EAC: Feature Model-based Enhanced Actor-Critic for Multi-Task Control in Dynamic Environments by Quanxi Zhou, Wencan Mao, Manabu Tsukada, John C.S. Lui, Yusheng Ji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15430.txt.
Introduction
Quanxi Zhou and co authors present FM-EAC in a 2025 paper on control systems that must handle many tasks in changing settings. Older methods tend to follow either a model based path or a model free path, and both can struggle to transfer skill across tasks. Their method aims to blend the strengths of the two.
FM-EAC, short for Feature Model based Enhanced Actor Critic, joins planning, acting, and learning in one framework. It uses new feature based models to improve how well control skill carries to new scenes. Tests in city and farm settings show it beating many leading methods from both families. Users can also swap in custom sub networks for their own needs.
Catalog record
| CREATOR | Quanxi Zhou, Wencan Mao, Manabu Tsukada, John C.S. Lui, Yusheng Ji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15430
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FM-EAC: Feature Model-based Enhanced Actor-Critic for Multi-Task Control in Dynamic Environments Quanxi Zhou, Wencan Mao, Manabu Tsukada, John C.S. Lui, Yusheng Ji Abstract Model-based reinforcement learning (MBRL) and model-free reinforcement learning (MFRL) evolve along distinct paths but converge in the design of Dyna-Q [ 1 ] . However, modern RL methods still struggle with effective transferability across tasks and scenarios. Motivated by this limitation, we propose a generalized algorithm, Feature Model-Based Enhanced Actor-Critic (FM-EAC), that integrates planning, acting, and learning for multi-task control in dynamic environments. FM-EAC combines the strengths of MBRL and MFRL and improves generalizability through the use of novel feature-based models and an enhanced actor-critic framework. Simulations in both urban and agricultural applications demonstrate that FM-EAC consistently outperforms many state-of-the-art MBRL and MFRL methods. More importantly, different sub-networks can be customized within FM-EAC according to user-specific requirements. K eywords Unmanned Aerial Vehicle, Reinforcement Learning, Actor Critic, and Feature Extraction. 1 Introduction Over the past few decades, it has been a highly debated topic of what the best approach is for decision-making, i.e., via planning or learning. Within a Markov decision process (MDP), where one or multiple agents interact with the environment, experience serves at least two key roles. First, it can be used to improve the model so that it can more accurately reflect the real environment - this is known as planning or model-based reinforcement learning (MBRL). Second, experience can be used directly to enhance the value function and policy - this is known as model-free reinforcement learning (MFRL) [ 1 ] . On the one hand, MBRL methods, including MBPO [ 2 ] , MORel [ 3 ] , COMBO [ 4 ] , CMBAC [ 5 ] , and Dreamer [ 6 ] , make use of a limited amount of experience and thus achieve a higher training efficiency with fewer environmental interactions. On the other hand, MFRL methods, including DQN [ 7 ] , DDPG [ 8 ] , PPO [ 9 ] , SAC [ 10 ] , and TD3 [ 11 ] , are much simpler and are not affected by biases in the design of the model, thus more suitable for complex environments. Despite the development on these two approaches, there exist many similarities between MBRL and MFRL methods, and such insights are reflected in the design of Dyna-Q [ 1 ] , which combines model-free learning with simulated experience from a learned model. However, a common limitation of existing MBRL and MFRL approaches lies in their lack of transferability . In the context of MBRL, a model of the environment (a.k.a., state transition model) means anything that an agent can use to predict how the environment will respond to its actions. Given a state and an action, a state transition model produces a prediction of the upcoming state and reward. Thus, MBRL methods are often tailored to specific models and environments. Unlike MBRL, MFRL methods utilize value and policy iterations to improve the optimality of the policies. The modern MFRL methods, especially those based on the actor-critic framework, perform well in complex tasks such as unmanned aerial vehicle (UAV) control [ 12 , 13 ] . Nevertheless, they are typically trained in fixed environments for individual tasks, which highlights the need for a generalized method capable of handling different tasks in dynamic environments. To this end, we envision a generalized model integrating planning, acting, and learning for multi-task control in dynamic environments. Our proposed algorithm, referred to as feature model-based enhanced actor-critic (FM-EAC), combines the advantages of MBRL and MFRL approaches while improving the generalizability and transferability. Real-world environments are featured by spatio-temporal variations. In this context, generalizability refers to the ability of the algorithm to adapt to various environmental conditions and task requirements, and transferability refers to the ability of the algorithm to remain robust when environmental conditions or task requirements change. Beyond existing methods, FM-EAC has two distinct modules: a feature model and an enhanced actor-critic framework. Despite the spatiotemporal variations in dynamic environments, it is possible to extract environmental features as representations and leverage them to train a feature model that is robust to environmental uncertainties. Such a feature model is combined with the state to enrich environmental information. Meanwhile, the actor-critic framework in MFRL offers higher flexibility for multi-task setups. Therefore, we propose an enhanced actor-critic framework to decouple the actors and critics for different tasks, enabling simultaneous policy updates. Figure 1: Overview of FM-EAC. The main contributions are summarized below: • FM-EAC, a feature model-based enhanced actor-critic algorithm, is proposed for multi-task control in dynamic environments. It combines the benefits of both MBRL and MFRL with improved generalizability and transferability . • Within FM-EAC, we demonstrate three exemplary sub-networks: graph neural network (GNN) [ 14 ] , point array network (PAN), and battery prediction network (BPN). GNN and PAN are adaptively trained networks and pre-trained frozen networks, respectively, for feature extraction of environments. BPN is a pre-trained frozen network for capacity prediction of batteries. • Different from existing reinforcement learning methods that are task- and environment-specific, our proposed FM-EAC can learn from various environments for multiple tasks, surpassing the state-of-the-art methods in performance, efficiency, and stability when environmental conditions or task requirements change. The code and results are publicly available through [link to demonstration code] . 2 Related Works Most existing research on generalized reinforcement learning algorithms has centered on meta-reinforcement learning, which trains a meta-policy on various tasks to embed prior knowledge and facilitate fast adaptation. Some research utilized recurrent neural networks (RNNs) to embed an agent’s learning process [ 15 , 16 ] . Others used gradient descent for policy adaptation in the inner loop [ 17 , 18 ] . However, the parameter tuning for the above methods requires intensive efforts, and the above methods increase generalizability at the cost of degrading performance. Inspired by few-shot learning in supervised tasks, feature metrics or external memory have also been used for policy adaptation. Our approach inherits such an idea, using a feature-based model to extract the feature representations that can rapidly adapt to various environments. Meanwhile, most existing research on multi-task control concentrated on discrete-continuous hybrid action spaces. P-DQN [ 19 ] is a parametrized DQN framework for the hybrid action space where discrete actions share the continuous parameters. Similarly, HD3 [ 20 ] , a distributed dueling DQN algorithm, was proposed to produce joint decisions by using three sequences of fully connected layers. Hybrid SAC [ 21 ] is an extension of the SAC algorithm, where an actor computes a shared hidden state representation to produce both the discrete and continuous distributions. HPPO [ 22 ] , a hybrid architecture of actor-critic algorithms, was proposed to decompose the action spaces along with a critic network to guide the training of all sub-actor networks. However, the above works considered an identical objective for all tasks, neglecting the interrelationship and heterogeneity between them. In contrast, FM-EAC can effectively handle interrelated tasks with their respective goals through the enhanced actor-critic framework. 3 Overview of FM-EAC FM-EAC, as illustrated in Fig. 1 , is composed of three main components: (1) the environment, (2) the enhanced actor-critic framework, and (3) the feature model. We demonstrate two