Improving Action Smoothness for a Cascaded Online Learning Flight Control System

Yifei Li, Erik-Jan van Kampen · 2025

Read Improving Action Smoothness for a Cascaded Online Learning Flight Control System by Yifei Li, Erik-Jan van Kampen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04346.txt.

Introduction

Li and van Kampen smoothed an adaptive flight controller in this paper from twenty twenty-five. Learning-based controllers can adapt in flight, but a twitchy one may command jerky moves. Jerks stress the actuators and can shake the aircraft. The target task is tracking a set angle of attack under changing dynamics.

Their scheme caps how fast control actions may change, with a penalty weight that adjusts itself online. A low-pass filter in the layered network trims high-frequency commands while keeping learning valid. Simulations on a longitudinal aircraft model showed smoother control and better tracking than the base method. Oscillations fell, and the trained policy stood up better to model errors. The authors stress the method's low computing cost.

Catalog record

CREATORYifei Li, Erik-Jan van Kampen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04346

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Action Smoothness for a Cascaded Online Learning Flight Control System Yifei Li, Erik-Jan van Kampen Abstract This paper aims to improve the action smoothness of a cascaded online learning flight control system. Although the cascaded structure is widely used in flight control design, its stability can be compromised by oscillatory control actions, which poses challenges for practical engineering applications. To address this issue, we introduce an online temporal smoothness technique and a low-pass filter to reduce the amplitude and frequency of the control actions. Fast Fourier Transform (FFT) is used to analyze policy performance in the frequency domain. Simulation results demonstrate the improvements achieved by the two proposed techniques. Index Terms - reinforcement learning, symmetry, data augmentation, approximate policy iteration, flight control. I INTRODUCTION the past decades, model-based control (MBC) design has played a major role in the field of flight control. With a well-calibrated model of real-world flight dynamics, including aerodynamic coefficients, actuator dynamics, the dynamic characteristics of aerial flight can be effectively represented and analyzed [ 1 ] . The modeling of aerodynamics offers benefits for flight control design in terms of interpretability, directionality, and effectiveness. Specifically, MBC design seeks to formulate the control law as the inverse of a model, thereby canceling the system dynamics. Representative examples include nonlinear dynamic inversion (NDI) [ 2 , 3 , 4 ] , backstepping (BS) [ 5 ] , and sliding mode control (SMC) [ 6 ] . However, these MBC methods are limited by model uncertainties, which increase the computational cost of model inversion and degrade control performance. To address this issue, uncertainty and disturbance estimation and attenuation techniques have been commonly employed alongside MBC. These methods estimate unmodeled dynamics and external disturbances by leveraging measured data, while simultaneously compensating through the control law [ 7 ] . Common examples include the disturbance observer (DOB) and extended state observer (ESO). Nevertheless, observer design also depends on a system model to compute estimation errors. In contrast, data-driven control techniques leverage system data to design and refine control laws, opening a broad design space for model-free control (MFC), which seeks to reduce reliance on system models. Although hybrid control approaches that integrate both system models and samples have improved the adaptability of model-based control, their capacity to optimize control performance is still a challenge. Examples of these methods are adaptive NDI [ 8 , 9 , 10 , 11 ] and adaptive backstepping [ 12 , 13 , 14 ] . A fully model-free control design is exemplified by model-free reinforcement learning (RL), a learning-based approach that minimizes reliance on system models [ 15 ] . In this framework, the control law can be parameterized as an Artificial Neural Network (ANN) and trained using system samples. The training process is viewed as policy optimization, guided by a critic network that approximates the value function. Another key component of the RL framework is policy evaluation, where the critic network is trained based on the principle of Temporal-Difference (TD) learning. In model-based reinforcement learning, a system model is still used to make one- or multi-step state predictions. However, model inaccuracies have been shown to cause only limited degradation in control performance [ 16 ] . The online implementation of RL, referred to here as online learning , focuses on learning from state and cost signals measured directly from a real system that cannot be reset. This implementation involves multiple updates to the critic and actor networks before executing an action on the physical system. Such an approach enables adaptive adjustment of the control law via gradient-based optimization, thereby expanding the applicability of RL to adaptive flight control, which deals with unpredictable model uncertainties. This has been investigated for both fixed-wing and rotor-wing unmanned aerial vehicles using a class of incremental model-based approximate dynamic programming methods that support online system identification and policy learning [ 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] . A representative example is Incremental Heuristic Dynamic Programming (IHDP) [ 19 , 24 , 25 ] , which employs an incremental model to linearly approximate the nonlinear system. This approach reduces both the model complexity and the computational burden of online learning. Notably, the incremental model does not need to be highly accurate; it is sufficient for the signs of the estimated values to be correct in order to support effective policy optimization [ 24 , 25 ] . Decomposing the dynamical system into reduced-order subsystems before control design can simplify the learning process in learning-based control frameworks. For instance, the pitch-rate dynamics is modeled as a first-order mapping from control surface deflection to pitch rate (see Equation 1 ). In this case, the value function associated with pitch rate tracking error and control effort is directly related to the actor network [ 26 ] . However, this design approach cannot be directly extended to angle-of-attack tracking, as the value function associated with tracking error fails to provide effective gradients for training control surface actor due to its indirect control effect on the angle of attack [ 1 ] . This requires a well-trained critic to provide accurate gradients, which is challenging in online learning due to limited sample availability. To address this, a hierarchical control strategy can be employed, separating the angle-of-attack and pitch-rate dynamics into two subsystems for independent control design [ 12 , 13 , 14 , 27 ] . A virtual pitch-rate command is first designed to control the angle of attack. This virtual command then serves as the reference for the pitch-rate tracking control system, resulting in a cascaded control structure. This design is reflected in many flight control applications [ 24 , 28 ] . The cascaded structure not only facilitates the integration of pitch-rate constraints but also simplifies the learning process by preventing the RL agent from using aggressive actions in the absence of such constraints. A practical example of this approach is the control of the F-16 Fighting Falcon using incremental global dynamic heuristic programming [ 20 ] . One key concern in upgrading cascaded online learning flight control systems is ensuring the smoothness of both virtual and actual control actions. Action smoothness significantly affects tracking performance and system stability [ 20 , 22 ] . A common strategy to promote smooth action transitions is the use of an action smoothness loss for actor training. The design of this loss depends on how action variation is defined, such as temporal or spatial variations [ 29 ] . While these methods have shown effectiveness in offline-trained control systems for quadrotors [ 29 ] , they have not yet been applied in online learning scenarios. In addition, introducing a low-pass filter between sub-control systems can help suppress high-frequency signals of the virtual control actions, thereby enhancing action smoothness. This approach has been successfully implemented in command-filtered INDI and backstepping [ 5 , 30 , 31 ] . However, the integration of a filter into a cascaded online learning flight control framework remains unexplored. The main contributions of this paper are summarized as • An optimization-based cascaded flight control system is developed to achieve angle-of-attack tracking. • A hybrid approach combining online temporal smoothness and a

Read the complete text