Dementia Insights: A Context-Based MultiModal Approach
Sahar Sinene Mehdoui, Abdelhamid Bouzid, Daniel Sierra-Sosa, Adel Elmaghraby · 2025
Read Dementia Insights: A Context-Based MultiModal Approach by Sahar Sinene Mehdoui, Abdelhamid Bouzid, Daniel Sierra-Sosa, Adel Elmaghraby (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01226.txt.
Introduction
Sahar Mehdoui and co-authors detect dementia from speech and writing together in this 2025 paper. Dementia erodes memory and daily function, and catching it early opens the door to care that slows the slide. Big pre-trained models for text and sound can spot the marks of decline, but most studies use one channel alone.
Their method fuses text and audio features, enriched with context, from leading pre-trained models. One pairing, a text model joined with an audio model, scored about eighty-three percent on the balanced test measure. Raw text beat hand-labeled expert data sets, a surprise with practical weight. The authors say such tools could screen cheaply, without costly expert labeling.
Catalog record
| CREATOR | Sahar Sinene Mehdoui, Abdelhamid Bouzid, Daniel Sierra-Sosa, Adel Elmaghraby |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01226
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dementia Insights: A Context-Based MultiModal Approach Sahar Sinene Mehdoui, Abdelhamid Bouzid, Daniel Sierra-Sosa, Adel Elmaghraby Abstract Dementia, a progressive neurodegenerative disorder, affects memory, reasoning, and daily functioning, creating challenges for individuals and healthcare systems. Early detection is crucial for timely interventions that may slow disease progression. Large pre-trained models (LPMs) for text and audio, such as Generative Pre-trained Transformer (GPT), Bidirectional Encoder Representations from Transformers (BERT), and Contrastive Language-Audio Pretraining (CLAP), have shown promise in identifying cognitive impairments. However, existing studies generally rely heavily on expert-annotated datasets and unimodal approaches, limiting robustness and scalability. This study proposes a context-based multimodal method, integrating both text and audio data using the best-performing LPMs in each modality. By incorporating contextual embeddings, our method improves dementia detection performance. Additionally, motivated by the effectiveness of contextual embeddings, we further experimented with a context-based In-Context Learning (ICL) as a complementary technique. Results show that GPT-based embeddings, particularly when fused with CLAP audio features, achieve an F1-score of 83.33 % 83.33\% , surpassing state-of-the-art dementia detection models. Furthermore, raw text data outperforms expert-annotated datasets, demonstrating that LPMs can extract meaningful linguistic and acoustic patterns without extensive manual labeling. These findings highlight the potential for scalable, non-invasive diagnostic tools that reduce reliance on costly annotations while maintaining high accuracy. By integrating multimodal learning with contextual embeddings, this work lays the foundation for future advancements in personalized dementia detection and cognitive health research. Keywords Dementia \cdot CHAT \cdot speech \cdot text \cdot Deep Learning \cdot Cross Attention \cdot GPT \cdot Bert \cdot CLIP \cdot CLAP \cdot embedding \cdot MLP \cdot In-Context Learning 1 Introduction Dementia is an increasing global health challenge, affecting millions of individuals and imposing significant burdens on healthcare systems World Health Organization (2023) . Early diagnosis is critical for facilitating timely interventions that can slow disease progression and improve patient outcomes. Although effective, traditional diagnostic methods such as neuroimaging and biomarker analysis are often costly, invasive, and inaccessible in many settings El Abiad et al. (2024) ; Chávez-Fumagalli et al. (2021) ; Zhao et al. (2023) ; Lee et al. (2017) . Consequently, research has increasingly focused on non-invasive, scalable diagnostic methods, particularly those utilizing speech and language analysis to detect cognitive impairments through linguistic and acoustic markers. Advancements in artificial intelligence (AI) and computational methods have significantly transformed dementia research. Early machine learning models, such as logistic regression and Random Forests, effectively identified cognitive decline using linguistic indicators such as coherence, hesitations, and repetitions ( Shah et al., 2021 ; Jahan et al., 2024 ) . The emergence of deep learning, particularly transformer-based architectures such as BERT Devlin et al. (2019) and GPT Radford et al. (2019) ; Brown et al. (2020) ; Adler et al. (2024) , has further improved the detection of subtle linguistic patterns associated with cognitive impairments. Specialized adaptations, like AD-BERT ( Mao et al., 2023 ) , have improved early dementia detection through pre-training on Alzheimer’s disease-related datasets. Large language models (LLMs), such as GPT- 4 Adler et al. (2024) , excel in few-shot learning and effectively capture nuanced linguistic expressions linked to cognitive decline Du et al. (2024) Beyond text-based analyses, audio data provide critical insights into dementia-related speech characteristics by examining features such as pitch, pauses, articulation changes, and prosody. Advanced signal processing techniques, including log-Mel spectrograms and delta coefficients, have proven to be effective in identifying cognitive impairments Meghanani et al. (2021) . Vision Transformers (ViTs) Dosovitskiy et al. (2021) have outperformed traditional CNN-based models in capturing global dependencies from speech signals ( Ilias et al., 2023 ) , further advancing audio-based dementia detection. Integrating text and audio data through multimodal approaches holds significant potential for improving classification accuracy. By combining the richness of textual features with complementary speech-based cognitive markers ( Han et al., 2022 ) , multimodal frameworks such as GP-Net, which Liu et al. (2022) proposed, have demonstrated superior performance over conventional transformer-based models in dementia detection ( Liu et al., 2022 ) . These integrated approaches underscore the value of harnessing symbolic annotations, linguistic structures, and acoustic properties in providing a more comprehensive assessment of cognitive health. Despite these advancements, several challenges persist. Many studies rely on structured clinical notes or specialized datasets, thereby limiting their applicability to real-world settings. Moreover, while transcribed speech data include valuable linguistic annotations, such as self-corrections and repetitions, their practical impact on machine learning performance remains underexplored. There is a critical need for dementia detection models that are accurate, scalable, generalizable, and interpretable. In this paper, we introduce a context-driven multimodal model as our primary contribution, integrating both text and audio features for dementia detection using large pre-trained models. Unlike prior studies, our method explicitly incorporates contextual information across both modalities, enhancing interpretability and robustness. Additionally, recognizing the importance of contextual learning in multimodal analysis, we explore an In-Context Learning (ICL) strategy to evaluate its potential as a complementary technique. Our experiments demonstrate that while ICL shows promise, our proposed framework significantly outperforms alternative methods, making it the core innovation of this study. Previous works, such as Pan et al. (2025) , have leveraged cross-attention mechanisms with self-supervised learning (SSL) models like wav2vec2.0 and transformer-based architectures such as BERT, but these primarily relied on structured datasets and emphasized audio features. This study addresses those limitations by integrating multimodal data with large language models and evaluating both expert-annotated and raw transcriptions to improve scalability and robustness. We leverage the best available pre-trained models in each modality, including CLIP OpenAI (2021) , GPT, and BERT for text embeddings, and CLAP Elizalde et al. (2023) for audio. Using the Pitt Corpus MacWhinney et al. (2011) , we conduct a broad comparative analysis, assessing the efficacy of expert-annotated versus raw datasets. Our findings offer valuable insights into how various text and audio features contribute to dementia detection, underscoring the potential of scalable, non-invasive diagnostic tools. The structure of this paper is as follows: section 2 provides an overview of existing dementia detection methodologies, including text-based, audio-based, and multimodal approaches. Section 3 describes the Cookie Theft Test Cummings (2019) and the Codes for the Human Analysis of Transcripts (CHAT) MacWhinney (2024) , both of which form the basis of our experimental framework. Section 4 describes the dataset utilized in this study. Section 5 presents the methodologies employed in this study. Section 6 reports the experimental results and their analysis. Finally, section 7 discusses the implications of our findings and proposes avenues for future research. 2 Related Work Dementia detection has advanced substantially with computational methods, offering innovative techniques for identifying cognitive decline Du et al. (2024) ; Mao et al. (2023) ; Han et al. (2022) ; Bouazizi et al. (2023) ; Ilias et al. (2023) ; Balagopalan et al. (2020) ; Zhu et al. (2021) ; Pan et al. (2025) ; Baevski et al. (2020) ; Haulcy and Glass (2021) ; BT and Chen (2024) . While traditional diagnostic tools such as neuroimaging and biomarkers remain effective, they are often invasive, expensive, and inaccessible in many settings. As a non-invasive, scalable alternative, speech and language analysis has emerged as a promising method for detecting cognitive impairments by examining text and audio changes indicative of dementia. This section reviews related work, categorized