Technical Report on Reinforcement Learning Control on the Lucas-Nülle Inverted Pendulum

Maximilian Schenke, Shalbus Bukarov · 2024

Read Technical Report on Reinforcement Learning Control on the Lucas-Nülle Inverted Pendulum by Maximilian Schenke, Shalbus Bukarov (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02264.txt.

Introduction

Maximilian Schenke and Shalbus Bukarov wrote a 2024 technical report on teaching a real inverted pendulum with reinforcement learning. The pendulum, a rod that must be swung up and balanced, is a classic control test. Their setup runs on teaching hardware made for engineering courses, with learning done on a separate computer from the real-time controller.

One design handles both the swing-up and the balancing act. The two computers talk over a vehicle-style bus link, and experiments on the rig are reported in detail. The report gives teachers a working recipe for bringing learning-based control into the lab.

Catalog record

CREATORMaximilian Schenke, Shalbus Bukarov
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-02264

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Technical Report on Reinforcement Learning Control on the Lucas-Nülle Inverted Pendulum Thanks: M. Schenke is a freelance engineer based in Paderborn, Germany, S. Bukarov is with the Lucas-Nülle GmbH in Kerpen, Germany. E-mail: [email protected], [email protected] Maximilian Schenke, Shalbus Bukarov Abstract The discipline of automatic control is making increased use of concepts that originate from the domain of machine learning. Herein, reinforcement learning (RL) takes an elevated role, as it is inherently designed for sequential decision making, and can be applied to optimal control problems without the need for a plant system model. To advance education of control engineers and operators in this field, this contribution targets an RL framework that can be applied to educational hardware provided by the Lucas-Nülle company. Specifically, the goal of inverted pendulum control is pursued by means of RL, including both, swing-up and stabilization within a single holistic design approach. Herein, the actual learning is enabled by separating corresponding computations from the real-time control computer and outsourcing them to a different hardware. This distributed architecture, however, necessitates communication of the involved components, which is realized via CAN bus. The experimental proof of concept is presented with an applied safeguarding algorithm that prevents the plant from being operated harmfully during the trial-and-error training phase. I Introduction This report highlights the key aspects of implementing a controller for the inverted pendulum problem as a reinforcement learning (RL) application with the use of Lucas-Nülle educational hardware. The inverted pendulum, being one of the standard problems of control theory and practice, is usually one of the first practical examples that students get to experiment with. Herein, its intuitiveness, replicability and smooth transition from nonlinear to linear dynamics earned it a special place in automatic control education. While a variety of established control methods such as linearized PID control [ 1 ] or model predictive control [ 2 ] are available for this nonlinear control plant, this article focuses the implementation of control via RL, which is a subdiscipline of machine learning that corresponds to optimal control. A schematic overview of the employed control concept is depicted in Fig. 1 . Fig. 1 : Schematic depiction of the closed-loop control system In the following, the discussed key points include the problem setting, the fundamentals of RL-based control and their application to the inverted pendulum. Targeting experimental validation, it is further investigated how to avoid infeasible operation during trial-and-error training (so-called safeguarding) and how the implementation of the overall algorithm on Lucas-Nülle hardware is realized. Finally, experimental results from training and application are presented. II Problem Setting The inverted pendulum is one of the most studied dynamical systems [ 3 ] . Despite being oftentimes viewed a toy example, it actually has a quite practical background, which is underlined by applications like the segway [ 4 ] or reusable launch vehicles (i.e., safely landing rocket stages) [ 5 ] . In this investigation, the swing-up and stabilization problem of the inverted pendulum is examined with use of the educational hardware from the Lucas-Nülle product portfolio. Herein, the basic scenario of a single pendulum rod on a cart is analyzed, whose schematic is depicted in Fig. 2 . Instead of a conventional, model-based control design procedure, the given control problem is tackled by means of RL. The controller design process and resulting controller performance comes with some characteristic traits that are different from conventional control methods: • The controller behavior is adapted in a learning phase. Herein, data from the plant system is collected within direct interaction, allowing consideration of comprehensive dynamics and parasitic effects as long as they are visible within the measurements. • Consequently, the controller is not dependent on system modeling. Especially, parameter values do not have to be available, and it is not necessary to identify them. • Making use of artificial neural networks (ANNs), a single, nonlinear control law is sufficient to handle both, the swing-up maneuver and the stabilization in the upper equilibrium 1 1 1 Contrary, linear controllers can only be used close to the equilibrium and conventional optimal controllers usually employ a gain scheduling approach wherein the swing-up is handled differently than the steady state. . • The learning phase is initialized with an untrained RL controller and, hence, control performance is usually insufficient at the beginning of the training and improves procedurally. This technical report targets to cover theory and implementation of the proposed approach, and will also discuss some experimental results. Fig. 2 : Conceptual depiction of the inverted pendulum Since the modeling and parameterization of the plant system is not of importance for the controller design, it is only briefly revisited for the readers convenience. The inverted pendulum on a cart can be described by a system of differential equations: \displaystyle\begin{split}F(t)&=(m_{\text{cart}}+m_{\text{rod}})\frac{\text{d}^{2}}{\text{d}t^{2}}x(t)-m_{\text{rod}}l\,\text{cos}(\theta(t))\frac{\text{d}^{2}}{\text{d}t^{2}}\theta(t)\\ &\qquad\qquad+m_{\text{rod}}l\,\text{sin}(\theta(t))\left(\frac{\text{d}}{\text{d}t}\theta(t)\right)^{2},\\ 0&=l\frac{\text{d}^{2}}{\text{d}t^{2}}\theta(t)-\cos(\theta(t))\frac{\text{d}^{2}}{\text{d}t^{2}}x(t)-g\sin(\theta(t)),\end{split} (1) with time t , linear position x and gravitational acceleration g . The angular displacement \theta takes herein a special role, as the main control goal is the minimization of the absolute displacement |\theta| . The mass of the cart and the applied force are denoted by m_{\text{cart}} and F , respectively. The pendulum rod carries a mass m_{\text{rod}} at its end while the rod itself is assumed weightless. The effective length l of the rod can be adjusted to allow experiments with different moments of inertia. Please note that frictional force is omitted within this model. The utilized inverted pendulum experimental rig from Lucas-Nülle comes with a readily-tuned speed controller. This can be seen in Fig. 1 , and allows simplification of the mechanical system model when assuming the speed control loop to be significantly faster than the angular movement of the pendulum. Hence, ( 1 ) collapses to \displaystyle\begin{split}\frac{\text{d}}{\text{d}t}x(t)&=v(t)\approx v^{*}(t),\\ \frac{\text{d}^{2}}{\text{d}t^{2}}\theta(t)&=\frac{1}{l}\cos(\theta(t))\frac{\text{d}}{\text{d}t}v^{*}(t)+\frac{g}{l}\sin(\theta(t)),\end{split} (2) with v^{*} denoting the speed reference, which herein takes the role of the manipulated variable by means of the RL controller. The momentary system state is specified by the state vector \bm{s} \displaystyle\bm{s}(t)=\begin{bmatrix}x(t)&v(t)&\theta(t)&\omega(t)\end{bmatrix}, (3) whereas \omega(t)=\frac{\text{d}}{\text{d}t}\theta(t) denotes the angular velocity. While the positional states x and \theta are directly measurable from the experimental rig, the velocity states v and \omega are not directly measured and need to be estimated on the basis of the available positional quantities. In the following, availability of corresponding estimate values \hat{v} and \hat{\omega} is assumed. The employed estimation method is presented in Appendix -A and subjects to the limiting assumption of no plant parameter knowledge. III Reinforcement Learning Control While the core of each RL control approach lies within the proper encoding of the control goal within the reward function, a brief overview over the applied algorithm, the deep deterministic policy gradient (DDPG) is to be presented first [ 6 ] . III-A Deep Deterministic Policy Gradient (DDPG) The goal of each RL application is the maximization of the return g , which is defined as the accumulated future reward: \displaystyle g_{k}=r_{k+1}+\gamma r_{k+2}+\gamma^{2}r_{k+3}+\gamma^{3}r_{k+4}+\dots. (4) Herein, the reward function r defines a rating of the momentary performance in terms of the control goal for each time step independently, and the discount factor \gamma\in[0,1[ ensures numerical existence of g and allows to prioritize between the near and the far future. Further, the action-value function q establishes a relation between the system observation \bm{o} , the applied action a and the

Read the complete text