AdaptiveCoPilot: Design and Testing of a NeuroAdaptive LLM Cockpit Guidance System in both Novice and Expert Pilots

Shaoyue Wen New York University, Michael Middleton* New York University & Northrop Grumman, Songming Ping Imperial college, Nayan N Chawla Virginia Tech, Guande Wu New York University, Bradley S Feest Northrop Grumman, Chihab Nadri University of Illinois, Urbana-Champaign, Yunmei Liu University of L · 2025

Read AdaptiveCoPilot: Design and Testing of a NeuroAdaptive LLM Cockpit Guidance System in both Novice and Expert Pilots by Shaoyue Wen New York University, Michael Middleton* New York University & Northrop Grumman, Songming Ping Imperial college, Nayan N Chawla Virginia Tech, Guande Wu New York University, Bradley S Feest Northrop Grumman, Chihab Nadri University of Illinois, Urbana-Champaign, Yunmei Liu University of L (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04156.txt.

Introduction

Shaoyue Wen and co-authors build a cockpit guide that adapts to a pilot's mental load, in work from 2025. Busy cockpits can overload pilots, which hurts performance. Their system reads workload from a light-based brain scan worn on the head.

A small study with expert pilots set rules for when to switch cue types and how much detail to show. The full system joins those rules with live behavior data and a language model that knows the flight context. Eight licensed pilots then flew a virtual cockpit under three setups. With the adaptive guide, pilots spent more time in an ideal load zone on memory and perception measures. The authors present it as a step toward load-aware flight aids.

Catalog record

CREATORShaoyue Wen New York University, Michael Middleton* New York University & Northrop Grumman, Songming Ping Imperial college, Nayan N Chawla Virginia Tech, Guande Wu New York University, Bradley S Feest Northrop Grumman, Chihab Nadri University of Illinois, Urbana-Champaign, Yunmei Liu University of L
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-04156

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AdaptiveCoPilot: Design and Testing of a NeuroAdaptive LLM Cockpit Guidance System in both Novice and Expert Pilots Shaoyue Wen New York University, Michael Middleton* New York University & Northrop Grumman, Songming Ping Imperial college, Nayan N Chawla Virginia Tech, Guande Wu New York University, Bradley S Feest Northrop Grumman, Chihab Nadri University of Illinois, Urbana-Champaign, Yunmei Liu University of Louisville, David Kaber University of Florida, Maryam Zahabi Texas AM University, Ryan P. McMahan Virginia Tech, Sonia Castelo New York University, Ryan Mckendrick** Northrop Grumman, Jing Qian** New York University, Cláudio T. Silva** New York University Abstract Pilots operating modern cockpits often face high cognitive demands due to complex interfaces and multitasking requirements, which can lead to overload and decreased performance. This study introduces AdaptiveCoPilot , a neuroadaptive guidance system that adapts visual, auditory, and textual cues in real time based on the pilot’s cognitive workload, measured via functional Near-Infrared Spectroscopy (fNIRS). A formative study with expert pilots (N=3) identified adaptive rules for modality switching and information load adjustments during preflight tasks. These insights informed the design of AdaptiveCoPilot , which integrates cognitive state assessments, behavioral data, and adaptive strategies within a context-aware Large Language Model (LLM). The system was evaluated in a virtual reality (VR) simulated cockpit with licensed pilots (N=8), comparing its performance against baseline and random feedback conditions. The results indicate that the pilots using AdaptiveCoPilot exhibited higher rates of optimal cognitive load states on the facets of working memory and perception, along with reduced task completion times. Based on the formative study, experimental findings, qualitative interviews, we propose a set of strategies for future development of neuroadaptive pilot guidance systems and highlight the potential of neuroadaptive systems to enhance pilot performance and safety in aviation environments. \vgtccategory Research \vgtcinsertpkg \teaser A . AdaptiveCoPilot is a neuroadaptive guidance system for pilot virtual reality preflight checklist training. B . Cognitive workload state is assessed through fNIRS neuroimaging classification models. C . AdaptiveCoPilot combines cognitive workload states, behavioral information, and carefully constructed adaptive rules into a context aware LLM. D . The Neural-Informed LLM reasons over the input to inform modality and information load adaptations with the goal of maintaining a pilots optimal workload. keywords Aviation in virtual reality, Adaptive user interface 1 Introduction \color blackModern cockpits have grown in both complexity and features, as such they require increasingly strong operational skill and knowledge [ 53 ] . Pilots often engage in multi-tasking which may saturate their cognitive abilities. Prolonged cognitive load saturation can also induce fatigue along with decreased performance, leading to errors and potential aircraft incidents [ 10 , 13 ] . Advancements in aviation training and guidance systems have not kept pace with the complexity of modern cockpits. Paper checklists are the primary form of aviation task guidance [ 18 ] and Crew Resource Management (CRM) training, although effective at reducing errors, has changed little since 1981 [ 29 ] . \color blackCRM training for aviation is defined as training to use hardware, software, and humans effectively to minimize aviation errors [ 34 ] . Mnemonic strategies such as “aviate–navigate–communicate” are taught to pilots in CRM courses as a hierarchical memorization technique to assign priorities and regulate workload in high-stress emergencies [ 47 ] . However, these strategies themselves require attentional resources. Further, the many step procedures found in aviation checklists [ 9 ] , are difficult to memorize through these techniques. \color blackRecently, brain-computer interfaces have shown potential in measuring and addressing areas of high stress and cognivive demand in aviation environments [ 63 , 19 ] . This research explores the benefits of AdaptiveCoPilot , an adaptive, multimodal feedback system implemented in a virtual reality (VR) cockpit that provides real-time guidance based on pilots’ cognitive states and performance. The system uses functional Near-Infrared Spectroscopy (fNIRS) to measure cognitive states across three facets—working memory, perception, and attention—and adaptively adjusts the modality and content of visual and auditory feedback. A formative study with three pilots \color blackfound the specific areas of high cognitive demands of aircraft operation and the challenges in selecting appropriate modality and content for feedback. The results from this study informed the development of neuroadaptive strategies aimed at maintaining optimal workload states across different cognitive facets. An eight-pilot case study provided initial evidence of the system’s effectiveness in managing cognitive load. Additionally, after completion of the study interviews with pilots from diverse backgrounds—including recreational, fighter, and expert Black Hawk pilots—offered valuable perspectives on the role of automated guidance and the benefits of the neuroadaptive system. This research makes the following contributions: • AdaptiveCoPilot , a system adapts the content and guidance modality via users’ cognitive states, providing visual, auditory, and text-based guidance with adaptive rules summarized from existing theories. • A case study on real pilots testing the efficacy of AdaptiveCoPilot through a VR-simulated preflight task, comparing baseline, random and adaptive conditions. Thus providing a functional proof-of-concept for generalized real-time neuroadpative guidance during pre-flight. • The insights gained from the qualitative evaluation are synthesized into strategies for designing and testing future AI-adaptive assistants. 2 Related Work We review the use of VR in high-fidelity simulations, methods to measure cognitive workload, and the development of neuroadaptive systems for adaptive feedback based on real-time physiological data. VR for Testing New Technology with High Ecological Validity. VR allows for the recreation of complex environmental contexts, offering immersive experiences that enhance ecological validity in testing and training scenarios [ 32 , 31 ] . In aviation, VR simulators provide pilots with realistic training environments where they can practice procedures and respond to simulated emergencies without the risks associated with real flights [ 1 ] . Similarly, VR has been used in military simulations to train personnel in realistic combat scenarios, improving readiness and decision-making skills [ 33 , 49 ] . This makes VR ideal for testing prototype experimental systems due to relative safety and its effectiveness. Measuring Cognitive Workload. Cognitive workload is the mental effort required to perform a task and is influenced by the interaction between task demands and an individual’s cognitive capabilities [ 60 , 23 , 46 , 56 ] . Our system focuses on cognitive capabilities in working memory, perception, and attention, as each of these cognitive facets has inherent processing limitations [ 57 ] . Since cognitive workload is not directly observable, it is commonly estimated using three primary methods. (1) Task Performance Measures: Evaluating performance on a primary or concurrent secondary task to infer cognitive load [ 46 ] . (2) Subjective Evaluations: Using self-reported assessments such as the NASA Task Load Index (NASA-TLX) [ 26 ] and the Modified Cooper-Harper Scale [ 30 ] . These methods can suffer from self-reporting biases and can often be misleading if not analyzed, accounting for individual differences in participant bias [ 44 ] . (3) Physiological Measures: Tracking biological responses such as respiration, eye movements, and brain signals which represent the underlying causal components of cognitive processing but are often noisy and difficult to interpret [ 15 ] . Real-Time Physiological Measurement of Workload. Physiological sensors and algorithmic classifiers can be used to provide unobtrusive estimates of cognitive workload in real-time. Monitoring cortical hemodynamics via the prefrontal cortex—critical for processing and control of

Read the complete text