Training Environment for High Performance Aircraft Reinforcement Learning

Gregory F. Search · 2025

Read Training Environment for High Performance Aircraft Reinforcement Learning by Gregory F. Search (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01953.txt.

Introduction

Gregory Search presents in 2025 an open-source training environment for reinforcement learning with high performance aircraft. Note that the text's own title and byline differ slightly from the catalog fields, and the text governs here. The tool, called Tunnel, embeds F16 flight dynamics in a standard Python gymnasium.

Tunnel offers building blocks for boundaries, targets, foes, and sensors that can vary by mission need. Planners can adapt quickly to new settings, and researchers gain realistic aircraft physics. A week-long trade study tested training methods, observation spaces, and threat setups. The author argues such agility can aid collaboration and national defense.

Catalog record

CREATORGregory F. Search
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01953

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training Environment for High Performance Aircraft Reinforcement Learning Gregory F. Search Abstract This paper presents Tunnel, a simple, open source, reinforcement learning training environment for high performance aircraft. It integrates the F-16’s 3D non-linear flight dynamics into OpenAI’s Gymnasium python package. The template includes primitives for boundaries, targets, adversaries and sensing capabilities that may vary depending on operational need. This offers mission planners a means to rapidly respond to evolving environments, sensor capabilities and adversaries for autonomous air combat aircraft. It offers researchers access to operationally relevant aircraft physics. Tunnel’s simple code base is accessible to anyone familiar with Gymnasium and/or those with basic python skills. This paper includes a demonstration of a week long trade study that investigated a variety of training methods, observation spaces, and threat presentations. This enables increased collaboration between researchers and mission planners which can translate to a national military advantage. As warfare becomes increasingly reliant upon automation, software agility will correlate with decision advantages. Airmen must have tools to adapt to adversaries in this context. It may take months for researchers to develop skills to customize observation, actions, tasks and training methodologies in air combat simulators. In Tunnel, this can be done in a matter of days. 1 Nomenclature AESA = Active Electronically Scanned Array BFM = Basic Fighter Maneuvers CCA = Collaborative Combat Aircraft CNN = Convolutional Neural Network CSAIL = Computer Science and Artificial Intelligence Lab DARPA = Defense Advanced Research Projects Agency EW = Electronic Warfare F-16 = Designation for the Fighting Falcon Aircraft FLCS = Flight Control System GPS = Global Positioning System LiDAR = Light Detection and Ranging LSTM = Long Short Term Memory LTC = Liquid Time Constant MIT = Massachusetts Institute of Technology ML = Machine Learning MLP = Multi-Layer Perceptron PID = Proportional Integrator Differentiator RNN = Recurrent Neural Network RWR = Radar Warning Receiver UAV = Unmanned Aerial Vehicle VENOM = Viper Experimental Next-Gen Operations Model VISTA = Variable In-Flight Stability Test Aircraft 2 Introduction F or the first time in over sixty years, the United States Air Force’s capability to achieve air superiority is at risk [ 1 ] . To mitigate this, Collaborative Combat Aircraft (CCA) are being designed with levels of autonomy never before seen in high performance air combat [ 2 ] . The state of the art is an F-16 capable of being controlled by an AI agent named the Variable In-Flight Stability Test Aircraft (VISTA). Though it can perform live air-air engagements, this unique aircraft design requires funds and time that are prohibitive to plans to deliver over one thousand CCA. Furthermore, VISTA does not integrate real world sensors, contested environments, non-cooperative adversaries or missions beyond air-air. The nation needs a way to rapidly discover capabilities and limitations of autonomous agents in a variety of air combat situations. This paper introduces Tunnel, a reinforcement learning environment created by the researcher. It is designed for simple modification to present the agent with a range of observations, actions, tasks and training methodologies. Currently, assessing autonomy in the high performance air domain is a slow process. The most recent VISTA flights have been in support of DARPA Air Combat Evolution (ACE) [ 3 ] . Teams undergo a rigorous build up from software in the loop simulation, then hardware in the loop simulation and constructive modeling prior to live flight. In 2022, the Department of the Air Force - Massachusetts Institute of Technology Artificial Intelligence Accelerator (DAF-MIT AI Accelerator, or DAF AIA) formed a team to participate in DARPA ACE. This team was able to train a novel class of algorithm, called a Liquid Time-constant Network [ 4 ] , to perform live autonomous flight within six months. This was less time than ACE training timelines, which can take years. Often times, teams must sacrifice exploration of algorithms to meet programmatic constraints imposed by this training timeline. Furthermore, the build-up to flight presented the teams with a fixed observation, action, task and training methodology. As autonomy evolves to handle real sensors, diverse mission environments and non-cooperative adversaries, these programmatic constraints may worsen if not properly mitigated. In the coming years, air combat autonomy will need to operate in far more challenging environments. The lessons learned from DARPA ACE are expected to be used to advance DARPA’s AI Reinforcements (AIR) project as well as the US Air Force’s Viper Experimental Next-Gen Operations Model (VENOM) program. AIR plans to research autonomous air combat with multiple agents while subjected to partial observability, concept drift and uncertainty [ 5 ] . VENOM plans to use operationally configured F-16 aircraft as high performance airborne test beds [ 6 ] . The intent is to use data ingested from operational sensors to construct an observation for the agent. 3 Related Work There are already simulations that can present AI agents with different air combat situations. There are also training environments that allow for rapid, diverse exploration of capabilities and limitations. Tunnel seeks to achieve the flexibility of training environments while maintaining a relevant representation of the high performance air domain. Some of these efforts are listed below: Drone Simulations - Airsim [ 7 ] and Pybullet-drones [ 8 ] are examples of training environments that model the behavior of small UAV. Some are effective at simulating multi-drone configurations. However, many don’t allow customization of the state, action and reward of the agent. Even if state, action and reward were to match, speed and maneuverability would likely make conclusions about high performance aircraft based on drone simulations invalid. DCS World and FSX - Many software products offer access to the flight dynamics of the F-16 and other high performance aircraft. Digital Combat Simulator (DCS) world [ 9 ] and Flight Simulator-X [ 10 ] are both low cost and offer mission customization. While users can make modifications, implementation of AI would typically be effects based. Meaning, the developer cannot alter the physics of the game directly. Proprietary Simulations - Defense primes such as Lockheed Martin, Northrop Grumman or Boeing own powerful, high fidelity simulations. These are prohibitive to most research teams due to distribution restrictions. Even for those allowed to use these products, there can be significant time and expertise required to learn to use the tool, let alone change the simulation environment. JSBSim and HarFang3D - JSBSim is an open source flight dynamics model. It leverages OpenAI Gymnasium and includes dozens of aircraft models [ 11 ] . Another example of open source simulation for high performance aircraft is HarFang3D, which features customizable air-to-air scenarios for reinforcement learning (RL) [ 12 ] . Both of these are written in C++. While these efforts have a strong body of documentation and tutorial videos, Tunnel should not require detailed instruction to use and is written using three files and less than 300 lines of code. MuJoCo & ALE - Open AI’s suite of RL environments have a strong diversity of tasks. MuJoCo challenges the agent to learn high dimensional observation and action spaces as a means to control the agents’ movements [ 13 ] . The Arcade Learning Environment (ALE) has hundreds of games designed to explore reward optimization in a breadth of contexts [ 14 ] . However, neither

Read the complete text