Cognitive Exoskeleton: Augmenting Human Cognition with an AI-Mediated Intelligent Visual Feedback Conference: Proceedings; 2023; Location CCS: Human-centered computing Interaction techniques CCS: Human-centered computing Interaction design CCS: Human-centered computing Ubiquitous and mobile computin

Songlin Xu, Xinyu Zhang · 2025

Read Cognitive Exoskeleton: Augmenting Human Cognition with an AI-Mediated Intelligent Visual Feedback Conference: Proceedings; 2023; Location CCS: Human-centered computing Interaction techniques CCS: Human-centered computing Interaction design CCS: Human-centered computing Ubiquitous and mobile computin by Songlin Xu, Xinyu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00846.txt.

Introduction

Time pressure can sharpen or wreck a person's focus, depending on the moment. Xu and Zhang build a system that adjusts time pressure on the fly to help people think better. A learning agent watches how a user is doing on math problems and changes the pressure to keep attention high without raising anxiety too far.

Training such an agent normally needs many costly user studies. The team instead trains two agents together: one sets the pressure, and a second pretends to be the user, copying behavior from an existing dataset. In a user study, people helped by the trained system did better than a baseline group. The authors present this paired training as a cheaper way to build feedback tools that adapt to a person's state in real time.

Catalog record

CREATORSonglin Xu, Xinyu Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00846

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cognitive Exoskeleton: Augmenting Human Cognition with an AI-Mediated Intelligent Visual Feedback Conference: Proceedings; 2023; Location CCS: Human-centered computing Interaction techniques CCS: Human-centered computing Interaction design CCS: Human-centered computing Ubiquitous and mobile computing systems and tools Songlin Xu, Xinyu Zhang Abstract . In this paper, we introduce an AI-mediated framework that can provide intelligent feedback to augment human cognition. Specifically, we leverage deep reinforcement learning (DRL) to provide adaptive time pressure feedback to improve user performance in a math arithmetic task. Time pressure feedback could either improve or deteriorate user performance by regulating user attention and anxiety. Adaptive time pressure feedback controlled by a DRL policy according to users’ real-time performance could potentially solve this trade-off problem. However, the DRL training and hyperparameter tuning may require large amounts of data and iterative user studies. Therefore, we propose a dual-DRL framework that trains a regulation DRL agent to regulate user performance by interacting with another simulation DRL agent that mimics user cognition behaviors from an existing dataset. Our user study demonstrates the feasibility and effectiveness of the dual-DRL framework in augmenting user performance, in comparison to the baseline group. Figure 1. Scenario demonstration of our dual DRL agents framework. The simulation DRL agent is first pre-trained from an existing dataset (1) to mimic human cognition behaviors. The regulation DRL agent could then be trained in the virtual environment by interacting with the simulation DRL agent (2) to learn potential time pressure control strategies for human cognition performance augmentation. The trained regulation DRL agent is then applied on real users (3) to improve their cognition performance using adaptive time pressure control strategies according to users’ real-time performance. Scenario demonstration of our dual DRL agents framework. The simulation DRL agent is first pre-trained from existing dataset (1) to mimic human cognition behaviors. The regulation DRL agent could then be trained in the virtual environment by interacting with the simulation DRL agent (2) to learn potential time pressure control strategies for human cognition performance augmentation. The trained regulation DRL agent is then applied on real users (3) to improve their cognition performance using adaptive time pressure control strategies according to users' real-time performance. 1. Introduction In spite of the giant leap-forward of AI technologies in the past decade, the human cognitive ability still far eludes machine systems. The situation will likely remain for long, since the current wave of AI resolution heavily relies on massive data, which differs from human cognition in principle, and is unlikely to surpass human’s ability of generalization and logical reasoning. To fundamentally overcome this limitation, AI can augment or even integrate with humans, instead of a replacement ( Cremer and Kasparov 2021 ) . One way to attain this vision is to develop intelligent feedback to improve human cognition performance. For example, by properly visualizing brain wave patterns to human users as a neurofeedback, the users could self-regulate their emotion status ( Faller et al. 2019 ) . Such neurofeedback needs to be judiciously presented, as a constant feedback may increase anxiety and reduce cognitive performance. Recent research ( e.g. , AttentivU ( Kosmyna and Maes 2019 ) and Dan et al. ( Szafir and Mutlu 2012 ) ) proposed to adapt the feedback over time to improve user engagement. When the system detects a drop of user engagement, it triggers corresponding feedback to regulate users immediately. However, this is a myopic feedback control strategy, and may not improve the overall performance across a cognitive process. Therefore, intelligent feedback to regulate user performance is a general problem that still needs further research. In this paper, we seek to answer a general question in realizing human-AI integrated systems, i.e. can we leverage AI models to augment and improve human cognition performance, through intelligent and continuous feedback? We explore the answers through a representative cognitive task, i.e. , math arithmetic problem solving (Sec. 3 ), which is widely used in evaluation of human cognition behaviors ( Lin et al. 2011 ; Judd and Klingberg 2021 ; Daitch et al. 2016 ; Costa et al. 2019 ) . We use time pressure as the feedback modality. Time pressure, presented through a visualization of time passing, has been proven to be effective in regulating cognition performance ( Cheng 2017 ; Slobounov et al. 2000 ; Moore and Tenney 2012 ) . It can increase the arousal of users ( Edland and Svenson 1993 ) by delivering a sense of urgency. However, a too strong time pressure may distract and stress users, thus degrading their performance. Existing research in cognitive science has demonstrated this trade-off ( Zur and Breznitz 1981 ; Slobounov et al. 2000 ) , but to our knowledge, no systematic solutions have been proposed and verified to adapt time pressure in real-time to continuously regulate human cognition. Ideally, an adaptive algorithm to control the existence of time pressure feedback according to users’ real-time performance can attain a sweet-spot. However, it is challenging to design such adaptation, as human cognition behaviors are highly dynamic and, unlike machine systems, do not follow closed-form models. Deep reinforcement learning (DRL) may automate such closed-loop adaptation. At a high level, improvement of user performance will result in larger reward for a DRL agent, which encourages it to learn the policies of adaptive time pressure that maximizes future reward, thus improving overall user performance throughout an entire cognitive task. However, DRL needs to be substantiated with a massive amount of training data. Even for well-defined math arithmetic tasks, it is infeasible to allow the DRL agent to interact with real users online to update its model, especially considering the slow paces of human cognitive tasks. In addition, whenever the hyperparameters change, the DRL model has to be retrained through a new user study. As a result, the iterative exploration of hyperparameters, which is taken for granted in machine perception tasks, becomes infeasible in such AI-mediated human cognitive tasks. Moreover, the initial exploration of the DRL agent training is usually random, which may jeopardize user performance. To tackle these problems, we propose a dual-DRL agent. Specifically, a regulation DRL agent (Fig. 1 ) controls the existence of time pressure feedback, while interacting with a simulation DRL agent that mimics real user cognition behaviors. Specifically, in the math arithmetic task, given specific math question and time pressure feedback as input, the simulation DRL agent could output corresponding user cognition performance, including answering accuracy and response time. The simulation DRL agent is pre-trained (Fig. 1 (1)) based on an existing dataset in our prior work ( Xu and Zhang 2023 ) . It serves as virtual users to train the regulation DRL agent (Fig. 1 (2)), thus overcoming the aforementioned human data crunch problem. For each math question, the regulation DRL agent decides whether time pressure should be delivered to virtual users according to their past performance. It will receive more reward if virtual users’ performance gets improved. This reward design encourages the regulation DRL agent to learn potential strategies to optimize virtual users’ performance. Upon convergence, the trained model can be used to regulate cognitive performance of real users (Fig. 1 (3)). With the dual-DRL agent design, the exploration capability of the DRL model can be massively enhanced as the

Read the complete text