Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech

Rez Samantha Z. Floresca, Edric Castel C. Hao, Hannah Grachiella Buñales, Chelsea Dominique E. Temprosa, Georgianna Z. Reyes, Kervin Gabriel L. Chua · 2026

Read Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech by Rez Samantha Z. Floresca, Edric Castel C. Hao, Hannah Grachiella Buñales, Chelsea Dominique E. Temprosa, Georgianna Z. Reyes, Kervin Gabriel L. Chua (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26007.txt.

Introduction

Floresca and five colleagues test speech-based dementia screening in Filipino and English. No earlier work had tried it in Filipino speech, though switching between the two languages is rife in the Philippines. The team builds a matched set of four thousand transcripts from DementiaBank, with hand-made Filipino versions that keep signs of mental decline. Five model types are tried under same-language training, zero-shot transfer, and joint training. The design splits language effects from task effects and shows where joint training gains most.

Catalog record

CREATORRez Samantha Z. Floresca, Edric Castel C. Hao, Hannah Grachiella Buñales, Chelsea Dominique E. Temprosa, Georgianna Z. Reyes, Kervin Gabriel L. Chua
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40496
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26007

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech Rez Samantha Z. Floresca, Edric Castel C. Hao, Hannah Grachiella Buñales, Chelsea Dominique E. Temprosa, Georgianna Z. Reyes, Kervin Gabriel L. Chua Abstract Dementia detection from spontaneous speech offers a scalable approach to cognitive screening, yet NLP systems remain predominantly English-centric. This limitation is especially acute in the Philippines, where Filipino–English code-switching is pervasive and no prior work has addressed NLP-based dementia detection. We present the first systematic evaluation of transformer-based dementia detection in Filipino speech and the first assessment of NeoBERT in a clinical NLP setting. To separate language from domain effects, we construct a parallel bilingual dataset of 4,000 DementiaBank-derived transcripts, with Filipino translations produced manually to preserve discourse-level markers of cognitive decline. We evaluate five model families, TF-IDF + LogReg, BERT, NeoBERT, XLM-R, and RoBERTa-Tagalog, under monolingual, zero-shot cross-lingual, and bilingual fine-tuning settings. We find that in-domain performance does not transfer across languages, with English-trained BERT dropping to Macro-F1 = 0.455 on Filipino, and that architectural modernization alone does not improve robustness. Bilingual fine-tuning, however, eliminates cross-lingual degradation across all transformer models, converging to Macro-F1 = 0.969–0.973. These results suggest that multilingual clinical NLP performance is driven primarily by linguistic coverage during training rather than model scale or architecture. 1 Introduction Dementia detection from speech is a promising direction for scalable cognitive screening, as linguistic and discourse-level patterns often reflect early cognitive decline 26 . However, most NLP systems for this task are developed and evaluated primarily on English-language data, limiting their robustness in multilingual and low-resource settings. This limitation is particularly relevant in the Philippines, where everyday speech frequently involves Filipino–English code-switching and informal conversational structure. As a result, it remains unclear whether failures arise from cross-lingual transfer limitations or English-centric biases in model pretraining, motivating our focus on conversational transcripts to isolate discourse-level markers of cognitive decline. Progress in Filipino NLP, while growing, has been largely confined to classification tasks on news articles and social media posts, including sentiment analysis 36 , named entity recognition 22 , and hate speech detection 7 . Even recent evaluation benchmarks such as Batayan 24 and FilBench 23 , while broader in scope, remain anchored to common written-text tasks such as reading comprehension, natural language inference, and generation. Critically, none of this work addresses naturalistic speech, clinical discourse, or cognitive diagnostic tasks. In particular, Tagalog–English code-switching, pervasive in informal and clinical Filipino communication 14 , is systematically absent from existing evaluations. As a result, it remains unknown whether transformer-based models can capture the discourse-level degradation characteristic of dementia in Filipino speech. Additionally, transformer-based encoders have become the dominant paradigm in clinical NLP 9 ; 25 , yet their application to dementia detection remains largely confined to architectural variants that differ only in pretraining data. ClinicalBERT 2 , AD-BERT 18 , and mBERT 27 all inherit the original BERT design unchanged. This leaves open the question of how architectural modifications affect the extent to which models capture cognitively diagnostic linguistic properties relevant to dementia detection. In this work, we study dementia detection under controlled bilingual conditions using conversational speech transcripts in English and Filipino. The full experimental setup is described in Section 3. Our contributions are threefold: (1) We present the first evaluation of NLP-based dementia detection in Filipino speech, using conversational transcripts manually translated from English under controlled domain conditions. (2) We provide the first empirical characterization of NeoBERT’s behavior under clinical and cross-lingual conditions. (3) We analyze cross-lingual transfer from English to Filipino under controlled conditions, benchmarking English-only models against multilingual (XLM-RoBERTa) and language-matched (RoBERTa-Tagalog) baselines to quantify the effect of English-centric pretraining under language shift. Our code is publicly available. 1 1 1 https://github.com/rezsam09/Filipino-English-Dementia-Classification 2 Related Work 2.1 Linguistic Markers of Cognitive Decline Spontaneous speech in dementia and mild cognitive impairment (MCI) exhibits reduced lexical diversity, increased disfluency, syntactic simplification, and degraded referential coherence, patterns that manifest measurably before clinical diagnosis 12 ; 31 and have been linked to early tau accumulation in language-relevant cortical regions 19 . A recent systematic review confirms that these markers are consistently recoverable from text alone, with linguistic-only models reaching an average diagnostic accuracy of 83% across 51 NLP studies of dementia detection 32 . However, existing work is heavily concentrated on monolingual English speech elicited under controlled clinical tasks, most commonly the Cookie Theft picture description task from DementiaBank 3 . Bilingualism introduces competing forces: lifelong code-switching is associated with delayed dementia onset through enhanced cognitive reserve 4 , while the syntactic mixing and lexical borrowing characteristic of Taglish may alter the observable linguistic signals on which current models rely 11 . This makes Filipino a theoretically significant yet entirely absent deployment context in the dementia NLP literature. 2.2 Transformer-Based Models in Clinical NLP In dementia detection, most systems adapt BERT through domain-specific pretraining: ClinicalBERT 2 adapts to clinical notes, AD-BERT 18 targets Alzheimer’s progression prediction, and mBERT 27 extends to multiple languages. These models differ mainly in pretraining data while sharing the same architecture, leaving open the question of whether architectural changes improve cross-lingual clinical performance. Representation pooling is another underexplored yet consequential design choice. Most existing systems default to [CLS] -token pooling, a choice optimized for the pretraining objective but not for aggregating task-relevant information across the full sequence. Reimers and Gurevych 28 demonstrate that mean pooling over final hidden states produces more stable and transferable sentence representations, particularly under domain and language shift, motivating our adoption of mean pooling across all transformer-based models. 2.3 Cross-Lingual Dementia Detection Cross-lingual transfer in dementia detection has been explored almost exclusively between typologically similar European languages. The ADReSS-M challenge 17 , including work by Tamm et al. 35 and 8 , established English-to-Greek transfer as a benchmark setting. Greek, however, shares significant Indo-European typological properties with English, limiting the generalizability of these findings to more distant transfer settings such as English-to-Filipino. Studies on dementia detection in low-resource, non-European languages remain rare. Adhikari et al. 1 applied NLP-based detection to English DementiaBank transcripts translated into Nepali, while Lu and Chen 16 extended this line of work to Amis, an indigenous Taiwanese language, finding that augmenting a small dataset of 80 scripts with translated English transcripts improved both robustness and accuracy. Neither study examines cross-lingual transfer nor evaluates multilingual pretraining strategies. No work has examined dementia detection in a Southeast Asian language or in any language spoken by a population where bilingualism and code-switching are the norm rather than the exception. Filipino, spoken by over 100 million people and characterized by pervasive Filipino–English code-switching 14 , represents exactly this gap. 2.4 Multilingual and Language-Matched Pretraining Multilingual pretraining has emerged as a common strategy for cross-lingual transfer in low-resource settings. XLM-RoBERTa 6 , trained on over 2.5 TB of filtered CommonCrawl data across 100 languages, substantially outperforms mBERT on low-resource cross-lingual benchmarks. However, even strong multilingual models do not fully capture the lexical and syntactic properties of Filipino: RoBERTa-Tagalog 7 , pretrained on TLUnified, a large-scale, thematically diverse Filipino corpus, demonstrates consistent gains over XLM-RoBERTa on Filipino classification benchmarks, suggesting that language-matched pretraining captures aspects of Filipino that multilingual models do not adequately model. Whether this advantage transfers to a

Read the complete text