Online Action-Stacking Improves Reinforcement Learning Performance for Air Traffic Control
Ben Carvell, George De Ath, Eseoghene Benjamin and Richard Everson · 2026
Read Online Action-Stacking Improves Reinforcement Learning Performance for Air Traffic Control by Ben Carvell, George De Ath, Eseoghene Benjamin and Richard Everson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04287.txt.
Introduction
Ben Carvell and three co-authors introduce online action-stacking, a wrapper that lets reinforcement learning policies issue realistic air traffic control instructions, in a 2026 arXiv paper. Training agents on tiny action sets is easier, but real controllers give compound clearances, not single nudges, so learned policies often speak the wrong language for the job.
Their agents train with simple heading or level steps plus a penalty that discourages chatter, so commands come in short bursts. At run time, action-stacking compiles each burst into one proper clearance. Trained with proximal policy optimisation in the BluebirdDT digital twin, the agents handle route following, climbs and descents. two-aircraft conflict avoidance, issuing far fewer instructions than a damped baseline while matching a policy trained on a much larger action space.
Catalog record
| CREATOR | Ben Carvell, George De Ath, Eseoghene Benjamin and Richard Everson |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-04287
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Online Action-Stacking Improves Reinforcement Learning Performance for Air Traffic Control Ben Carvell, George De Ath, Eseoghene Benjamin and Richard Everson Abstract We introduce online action-stacking , an inference-time wrapper for reinforcement learning policies that produces realistic air traffic control commands while allowing training on a much smaller discrete action space. Policies are trained with simple incremental heading or level adjustments, together with an action-damping penalty that reduces instruction frequency and leads agents to issue commands in short bursts. At inference, online action-stacking compiles these bursts of primitive actions into domain-appropriate compound clearances. Using Proximal Policy Optimisation and the BluebirdDT digital twin platform, we train agents to navigate aircraft along lateral routes, manage climb and descent to target flight levels, and perform two-aircraft collision avoidance under a minimum separation constraint. In our lateral navigation experiments, action stacking greatly reduces the number of issued instructions relative to a damped baseline and achieves comparable performance to a policy trained with a 37-dimensional action space, despite operating with only five actions. These results indicate that online action-stacking helps bridge a key gap between standard reinforcement learning formulations and operational ATC requirements, and provides a simple mechanism for scaling to more complex control scenarios. 1 Introduction 1.1 Background A utomation of Air Traffic Control represents one of the most critical challenges in aviation. The International Air Transport Association (IATA) estimates that 5.2 billion passengers will fly in 2025 [ 1 ] , with this figure projected to increase by up to 12.4 billion by 2050 [ 2 ] . The UK Government has matched the IATA “Fly Net Zero” resolution with their own “Jet Zero” commitment to net zero carbon emissions for aviation by 2050 [ 3 ] . In this context, the need for advanced decision support systems that can revolutionise operational efficiency becomes increasingly urgent. Meeting these demands while maintaining stringent safety standards requires domain-relevant research that establishes robust foundations for Air Traffic Control Officer (ATCO) automation support. ATCOs in the UK utilise a range of support systems to assist them in the controlling task, but the sophistication of these systems varies. The most advanced are predictive systems such as iFACTS [ 4 ] and iTEC [ 5 ] , which provide forecast predictions of aircraft behaviour driven by the Base of Aircraft DAta (BADA) model of aircraft performance developed by Eurocontrol [ 6 ] . These tools then compare these predictions to aid in identifying potential conflicts between aircraft, and to facilitate “what-if” probing of potential clearances to assess their ramifications. No suggestion of what clearance to issue is given, and responsibility for decisions and their safety rests entirely with the ATCO. Research to move towards full decision automation in ATC has been ongoing since the work of Wesson et al. in the 1970s [ 7 ] , with a rich history of approaches studied and prototyped. Rules-based and heuristic approaches have been well documented, with the most mature example being ARGOS, a prototype system developed at Eurocontrol [ 8 ] . Significant industry efforts have been made within the SESAR (Single European Sky ATM Research) programme, including AGENT [ 9 ] on rules-based approaches to conflict resolution, and more recently HYPERSOLVER [ 10 ] using reinforcement learning to manage both flow management and tactical ATC simultaneously. Reinforcement Learning (RL) has emerged as a promising approach to this challenge, with numerous studies exploring its application to ATC tasks [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] . Unlike traditional rules-based systems, RL offers the capacity to learn from experience and discover novel solutions to complex scenarios, as demonstrated in the groundbreaking work of DeepMind on AlphaStar and AlphaGo [ 19 , 20 ] , OpenAI on Dota 2 [ 21 ] , and in other applications to domains such as robotics [ 22 ] . These characteristics make RL particularly suitable for the dynamic and nuanced decision-making required in air traffic control. 1.2 Reinforcement Learning and PPO In RL, an agent is trained to maximise the cumulative reward acquired by choosing actions to take within a given environment [ 23 ] , with this interaction formulated as a Markov Decision Process [ 24 ] . The agent interacts with the environment over a series of discrete time steps \mathit{t}=0,1,2,\dots and at each time step must select an action \mathit{A}_{t} from a set of available actions \mathcal{A} given the current observed state of the environment \mathit{S}_{t}\in\mathcal{S} . The representation of the state must be chosen such that Pr[ \mathit{S}_{t+1}\,|\,\mathit{S}_{t},\mathit{A}_{t} ] = Pr[ \mathit{S}_{t+1}\,|\,\mathit{S}_{1},\mathit{A}_{1},\dots,\mathit{S}_{t},\mathit{A}_{t} ] to satisfy the Markov property, i.e., the probability of transitioning to the successor state \mathit{S}_{t+1} is independent of all previous states and actions except for the current state \mathit{S}_{t} and action \mathit{A}_{t} . As shown in Figure 1 , the environment then advances one time step, and the agent receives a reward \mathit{R}_{t+1} and an updated observation of the state \mathit{S}_{t+1} , forming a tuple of state, action, the resulting reward, and the next state \langle\mathcal{S}_{t} , \mathit{A}_{t} , \mathit{R}_{t+1} , \mathit{S}_{t+1}\rangle . Figure 1 : The interaction loop of agent with environment in a Markov Decision Process. By processing the trajectory of these tuples through time over many repeated training episodes, an agent can learn which actions to choose given a specific state to maximise the cumulative reward it receives. Agent behaviour is modulated through a discount factor \gamma\in(0,1] , which controls how much an agent prefers to receive immediate reward rather than predicted future reward. A low value of \gamma will result in a preference for immediate reward, whereas a high \gamma will result in the potential sacrifice of immediate reward for the chance of higher total reward later. There are two primary methods for training agents, policy-based and value-based methods. Policy-based methods learn a policy \pi ( \mathit{s} ) which maps any given state to a resultant action (or probability of choosing actions), while value-based methods learn a value function \mathit{V}^{\pi} ( \mathit{s} ) which predicts the expected return (total reward from the current time step until the end of an episode) given a particular state when following a policy [ 23 ] . For this work, we use Proximal Policy Optimisation (PPO) [ 25 ] , a state-of-the-art reinforcement learning method that combines elements of the two aforementioned families of methods, in a configuration known as actor-critic. In this configuration, both a policy \pi and a value function \mathit{V}^{\pi} are trained, with the value function then being used as the “critic”. By recording the actual total reward accrued under a policy, and then comparing this with the expected return given by the value function, a measure of how much better or worse an agent is performing against prior experience is obtained, with this used to guide updates to the policy. Combined with an efficient method for limiting the magnitude of policy updates to prevent instability in training, PPO has become widely regarded as a best-in-class method, being used for DeepMind’s Alphastar [ 19 ] and for refining output in recent groundbreaking Large Language Models [ 26 ] . 1.3 Prior Work: RL Formulations of Air Traffic Control The suitability of reinforcement learning for solving sequential decision-making