ASR-Agnostic Multimodal Spectrotemporal Modeling for Early Dementia Detection
Ugwu Chukwuemeka, Richard Oluwafemi Oyeleke · 2026
Read ASR-Agnostic Multimodal Spectrotemporal Modeling for Early Dementia Detection by Ugwu Chukwuemeka, Richard Oluwafemi Oyeleke (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30646.txt.
Introduction
Ugwu Chukwuemeka and Richard Oluwafemi Oyeleke build a dementia screening tool that listens to speech rather than reading transcripts. Speaking draws on the same memory and attention skills used in daily life, so speech can signal cognitive decline early. Most existing tools need automatic transcription first, throw away timing patterns, and were tested on one English corpus with known recording flaws.
Their framework works directly on Mel spectrograms, pictures of sound energy over time and pitch. It tracks how energy patterns shift between frames, fuses those cues with learned audio embeddings through cross-attention, and pools them with a transformer encoder. Separate models trained on English, Slovak and Spanish clinical corpora behaved very differently. The Slovak model reached about eighty-four percent accuracy, the Spanish one scored moderately, and the English one stayed near chance, confirming its corpus flaws. Fusion helped only when signal was spread across feature types.
Catalog record
| CREATOR | Ugwu Chukwuemeka, Richard Oluwafemi Oyeleke |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30646
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ASR-Agnostic Multimodal Spectrotemporal Modeling for Early Dementia Detection Ugwu Chukwuemeka, Richard Oluwafemi Oyeleke Abstract Speech production recruits the same executive, attentional, and working memory processes that underpin instrumental activities of daily living (IADLs), making it a compelling non-invasive proxy for functional cognitive assessment. Yet most speech-based dementia detection systems depend on transcription, discard within-recording temporal structure, and are validated on a single English corpus with documented recording artifacts. We propose an ASR-agnostic framework that operates directly on Mel spectrograms. Our key contribution is the extraction of spectrotemporal displacement fields from consecutive spectrogram frames, capturing how spectral energy patterns shift and fluctuate as digital biomarkers of cognitive decline. These features are fused with CNN–ConvGRU acoustic embeddings through a learned cross-attention mechanism and aggregated via a Transformer encoder with learnable query pooling. A composite temporal loss enforces smoothness and contrastive coherence across segments. We train three independent models on English (DementiaBank), Slovak (EWA-DB), and Spanish (Ivanova) corpora, each grounded in clinical cognitive elicitation protocols that tax IADL-relevant cognitive domains. The Slovak model achieves 83.9% accuracy (F1 = 0.878), Spanish achieves AUC = 0.788, while the English baseline yields 53.2%, confirming known corpus artifacts. Ablation studies across all three languages reveal three distinct fusion regimes: removing cross-attention collapses Spanish performance to 53.7% (below either unimodal model), while on Slovak the audio encoder alone outperforms the full model (93.7% vs. 83.9%), and all English configurations remain near chance. This demonstrates that the value of multimodal fusion is corpus-dependent: essential when signal is distributed across modalities, counterproductive when one modality dominates, and irrelevant when no signal exists. Auxiliary temporal losses converge to language-invariant values (feature CV = 3.4%), providing evidence of cross-lingual architectural stability. 1 Introduction Dementia affects over 55 million people worldwide, with cases projected to reach 153 million by 2050 [ 28 ] . Diagnosis currently depends on amyloid PET imaging, cerebrospinal fluid analysis, and multi-hour neuropsychological batteries, all of which are expensive, invasive, or inaccessible. Up to 75% of dementia cases remain undiagnosed globally, rising to 90% in low- and middle-income countries [ 41 , 23 ] . This diagnostic gap has motivated interest in speech as a non-invasive screening modality [ 17 , 32 ] . The clinical logic is compelling: speech production is not merely a language task but a complex cognitive act that simultaneously recruits executive function, working memory, attention, and self-monitoring, the same domains whose decline defines impairment in instrumental activities of daily living [ 25 , 33 ] . Verbal fluency, for instance, correlates more strongly with observation-based functional tests than any other cognitive measure in mild dementia [ 33 ] , and connected speech decline has been shown to precede clinical MCI diagnosis [ 27 ] . The elicitation tasks used in our three evaluation corpora (picture description, structured naming, reading aloud) are specifically designed to tax these overlapping cognitive demands, positioning speech analysis as a continuous, ecologically valid proxy for IADL-relevant function [ 18 , 9 ] . Three limitations constrain the current landscape. First, the majority of systems depend on automatic speech recognition (ASR) or manual transcription to extract linguistic features [ 35 ] , introducing a dependency on transcription quality that degrades across languages and recording conditions. Purely acoustic approaches avoid this but plateau at 70–82% accuracy [ 12 ] . Multimodal fusion has pushed results to 83–93% [ 14 , 21 ] , yet naive fusion provably hurts : modalities compete under joint training, and the weaker modality may never be discovered [ 40 , 13 ] . Second, most systems aggregate features across an entire recording, discarding within-utterance temporal evolution. Dynamic modeling outperforms static aggregation by roughly 8 percentage points [ 37 ] , and clinical evidence confirms that dementia manifests as temporal instabilities in jitter, shimmer, pitch contour, and speech rhythm [ 30 , 26 ] . Third, 63% of speech-based AD studies rely on DementiaBank [ 35 ] , a corpus in which classification models achieve near-perfect accuracy using only silent segments, exploiting recording artifacts rather than speech biomarkers [ 22 ] . We address all three limitations through a unified framework with four contributions: 1. Spectrotemporal displacement fields. We compute dense 2D displacement fields between consecutive spectrogram frames, capturing formant trajectory deviations, pitch instability, and energy redistribution as digital biomarkers. This extends the spectrogram-as-image paradigm [ 11 ] and 1D audio flow [ 8 ] to full 2D spectrotemporal analysis. 2. Cross-attention multimodal fusion. Spectral dynamics features are fused with CNN–ConvGRU acoustic embeddings through learned cross-attention, preventing the destructive modality interference documented under naive combination [ 13 ] . 3. Temporal regularization. A composite loss enforces smoothness, contrastive coherence, progression regularity, multi-scale consistency, and attention sparsity across speech segments. 4. Trilingual validation with complete ablation. Three independent models trained on English, Slovak, and Spanish corpora whose elicitation protocols tax IADL-relevant cognitive domains. Full ablation on all three languages reveals three distinct fusion regimes, establishing that the value of cross-attention is corpus-dependent rather than universally beneficial. 2 Related Work 2.1 Speech as a Cognitive and Functional Biomarker The theoretical basis for speech-based dementia detection rests on the shared cognitive architecture between speech production and daily functional competence. Levelt’s model decomposes speech into conceptualization, formulation, and self-monitoring [ 20 ] , each requiring the executive, attentional, and memory resources that IADLs also demand [ 16 ] . Marshall et al. showed that executive dysfunction independently predicts IADL decline even after controlling for memory [ 25 ] , and connected speech has been validated as a marker of disease progression in autopsy-confirmed AD [ 2 ] . De la Fuente Garcia et al. reviewed 51 studies and identified the Pitt Corpus as the field’s most heavily used resource, while arguing for ecologically valid monitoring paradigms [ 6 ] . Recent work has begun validating remote, home-based speech collection with associations to amyloid pathology [ 39 ] and longitudinal cognitive tracking [ 36 ] . 2.2 ASR-Agnostic and Acoustic Approaches Agbavor and Liang achieved AUC of 0.846 using data2vec without any transcription [ 1 ] . Haider et al. showed that paralinguistic features alone reach 78.7% accuracy through hard fusion [ 12 ] . The ADReSS-M Challenge confirmed that speech timing features (pause duration, speech rate) transfer across languages with AUC = 0.75, while lexical features fail [ 24 , 31 ] . These results motivate our fully ASR-agnostic design, operating on Mel spectrograms without transcription or language models. 2.3 Multimodal Fusion and Its Failure Modes Multimodal dementia detection has shown promise: Chu et al. achieved AUC of 93.0% with audio-visual fusion [ 4 ] , and Lee et al. used Shapley-value weighting across three modalities for 90.6% accuracy [ 19 ] . However, Wang et al. demonstrated that naive late-fusion networks underperform unimodal baselines [ 40 ] , Huang et al. proved this modality competition theoretically [ 13 ] , and Du et al. showed that jointly trained encoders learn worse representations than independent ones [ 7 ] . Our cross-attention mechanism addresses this by enabling selective, temporally-aligned information exchange rather than blind combination. Importantly, all prior multimodal systems require separate sensor streams; our framework derives both modalities from the same audio signal. 2.4 Temporal Modeling and Spectrotemporal Analysis Pan et