Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions

Angela Lopez-Cardona 1 2 , Sebastian Idesis 1 , Ioannis Arapakis 1 · 2025

Read Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions by Angela Lopez-Cardona 1 2 , Sebastian Idesis 1 , Ioannis Arapakis 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06843.txt.

Introduction

Angela Lopez-Cardona, Sebastian Idesis, and Ioannis Arapakis review work that feeds human cognitive signals into language models in a 2025 survey. Eye tracking and related measures record how people read, offering a trace of attention and processing effort. Adding such signals to model training promises help with scarce data and the heavy cost of training large models.

The review covers language models and multimodal systems, sorting advances and applications across the young field. Cognitive signals can guide models toward human-like reading patterns and flag hard passages. The authors close with open problems and directions, including richer signals and fairer evaluation of the claimed gains.

Catalog record

CREATORAngela Lopez-Cardona 1 2 , Sebastian Idesis 1 , Ioannis Arapakis 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06843

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions. Thanks: © 2025 IEEE. This is the author’s accepted manuscript. The final version will appear in IEEE under the Proceedings of the International Joint Conference on Neural Networks 2025. Angela Lopez-Cardona 1 2 , Sebastian Idesis 1 , Ioannis Arapakis 1 Abstract Recently, the integration of cognitive neuroscience in Natural Language Processing (NLP) has gained significant attention. This article provides a critical and timely overview of recent advancements in leveraging cognitive signals, particularly Eye-tracking (ET) signals, to enhance Language Models (LMs) and Multimodal Large Language Models (MLLMs) . By incorporating user-centric cognitive signals, these approaches address key challenges, including data scarcity and the environmental costs of training large-scale models. Cognitive signals enable efficient data augmentation, faster convergence, and improved human alignment. The review emphasises the potential of ET data in tasks like Visual Question Answering (VQA) and mitigating hallucinations in MLLMs , and concludes by discussing emerging challenges and research trends. I Introduction Language Models (LMs) have emerged as powerful tools for processing textual representations and modeling complex relationships between input features and output labels. Their applications span a wide range of tasks, including text generation, comprehension, translation, and sentiment analysis [ 1 , 2 ] . These developments in Natural Language Processing (NLP) have been significantly accelerated by Transformer architectures [ 3 ] , which enabled the parallelization of input sequence processing via self-attention mechanisms that capture long-range dependencies effectively. Building on the foundation of Transformer architectures, models like BERT [ 2 ] were proposed, leveraging Transformer encoders to capture bidirectional context within text and popularising the pretraining and fine-tuning paradigm. Similarly, Transformer decoders led to the development of models like GPT [ 1 ] , which framed language modeling as an autoregressive task. These advancements contributed to the rapid rise of Large Language Models (LLMs) , marking a milestone in NLP . Popular examples such as GPT-4 [ 4 ] and LLaMA 3 [ 5 ] have become benchmarks in Artificial Intelligence (AI) , establishing the term “generative language models” due to their ability to produce coherent and human-like responses. More recently, LLMs have evolved into Multimodal Large Language Models (MLLMs) , which extend their functionality beyond textual data, to train over heterogeneous datasets including images, audio, and video. This enables MLLMs to process, understand, and generate content across various formats, further bridging the gap between data representations [ 6 ] . An emerging challenge in this context is the scarcity of sufficient data or the suboptimal use of existing data collections for training, which creates a significant bottleneck [ 7 ] . Current estimates suggest that the available stock of public human-generated text data is reaching saturation [ 8 ] . This calls for rethinking our training paradigm and researching effective synthetic data generation methods. Another important issue is the significant environmental cost associated with training large-scale models. As the size and complexity of LLMs and MLLMs continue to expand exponentially, their energy demands have surged, resulting in substantial CO 2 emissions. For example, while training the original Transformer model [ 3 ] resulted in an estimated emission of 0.012 tonnes of CO 2 , the training of GPT-3 generated approximately 502 tonnes [ 9 , 10 , 11 ] , a drastic increase across four orders of magnitude. These statistics highlight the pressing need for more sustainable and energy-efficient training practices. In addition to these considerations, aligning models with human values—such as the ability to follow instructions, avoid harmful behaviors, and engage in safe, user-friendly interactions—remains a significant challenge [ 12 , 13 ] . Moreover, there is the persistent issue of models hallucinating, which leads to the generation of factually incorrect content [ 6 , 13 ] . A promising research direction is to enrich existing datasets with novel representations and user-derived cognitive signals. This approach can augment current data sources, reduce training data requirements, and enable faster convergence with more efficient learning for LLMs . Moreover, such data augmentation methods hold the potential to improve human alignment and reduce hallucinations. One representative example is Eye-tracking (ET) , which allows the study of visual attention and behaviour in a non-invasive and accurate manner. Reading is a complex process of information foraging that engages numerous cognitive functions [ 14 ] , [ 15 ] . A substantial body of research has explored oculomotor behaviour in the context of reading comprehension and broader reading tasks, offering valuable insights into how attention patterns relate to human preferences and reward modeling [ 16 ] . For example, [ 17 ] demonstrated that oculomotor behavior reflects precision-weighted prediction error minimization within the framework of active inference. Similarly, models like UniAR [ 18 ] further highlight how visual attention patterns can predict explicit human feedback. The present article provides a focused and timely review of the most recent research that incorporates cognitive signals in language modelling and understanding, with a particular emphasis on the use of ET as a proxy for cognitive processes. Given the emergence of linguistic applications that require images, such as Visual Question Answering (VQA) , and the integration of vision and language understanding in MLLMs , we also consider the application of cognitive signals in image processing. However, a comprehensive review of the state of the art on LLMs and MLLMs extends beyond our scope, and for that we refer the readers to [ 12 ] and [ 6 ] , respectively. The structure of this article is organised as follows. First, in Section II , we provide a comparative analysis of our work with existing reviews, highlighting key distinctions. In Section III , we examine the modelling of various data modalities ( Figure 1 .2), followed by an exploration of their diverse applications in Section IV ( Figure 1 .3A). Next, we discuss methods of integration in Section V and their potential role in explaining LMs in Section VI ( Figure 1 .3B). Our goal is to provide a focused guide to recent advancements in this field, while addressing current challenges ( Section VII ) and outlining promising directions for future research ( Section VIII ). Fig. 1: Schematic overview illustrating the use of cognitive signals (2) to enhance LMs (3A) and to explore similar representations (3B). (a) : applications are discussed in Section IV , (b) : integration methods in Section V . II Related work In [ 19 ] , the authors review research that utilises various types of cognitive signals acquired during language understanding tasks, focusing on their interplay with Machine Learning (ML) -based NLP methods. Their work discusses both ET and neuroimaging techniques, whereas the present article narrows the focus specifically to ET . Other related reviews include [ 16 ] and [ 20 ] , which explore alternative approaches to the costly and time-intensive data acquisition of gaze behavior at runtime. In contrast, our work examines only the most recent research, reflecting the rapid evolution of the field and the need for an up-to-date review that aligns with broader developments in the field. Last, in [ 21 ] , the authors provide an overview of recent efforts to incorporate human attention mechanisms into Deep Learning (DL)

Read the complete text