ParkMAE: a cross-linguistic masked autoencoder framework for robust Parkinson’s disease detection from speech Authors: Angelika Ando, Adrien Lesage, Marc De Gennes, Rachid Riad Published: 2025 ABSTRACT AbstractWe introduce ParkMAE, a robust multilingual speech foundation model and comprehensive benchmarking system for Parkinson’s disease assessment. We curated multiple large-scale speech datasets comprising approximately 750 h of pretraining data and 100 h of evaluation data across four languages and diverse clinical populations. Our self-supervised masked autoencoder approach, pretrained on this multilingual corpus, demonstrates superior performance achieving 39% F1 score for cross-linguistic diagnosis, outperforming existing acoustic markers (eGeMAPS) by a significant margin and maintaining comparable performance to generic speech models (Whisper), while using 89% fewer parameters. Besides, ParkMAE shows exceptional generalizability to unseen languages without language-specific finetuning. Beyond diagnosis, we systematically evaluate medication state monitoring and disease staging tasks, revealing that despite promising literature reports, current publicly available datasets and speech-based approaches fail to reliably capture these clinical dimensions. For cognitive assessment (MoCA), our model demonstrated predictive capability (F1 = 0.56), suggesting potential for speech-based cognitive monitoring. This comprehensive evaluation establishes both the capabilities and current limitations of speech-based Parkinson’s disease assessment, providing a reproducible framework for future clinical development.Keywords: Parkinson’s disease, Cross-linguistic, Self-supervised learning, Speech, Masked autoencodersSubject terms: Computational biology and bioinformatics, Health care, Mathematics and computing, Medical research, Neurology, Neuroscience Introduction Parkinson’s Disease (PD) is a progressive neurodegenerative disease affecting approximately 6.1 million people worldwide1. It manifests with a broad range of motor and non-motor symptoms that evolve over decades, substantially reducing quality of life and imposing significant burdens on patients, caregivers and society2. While motor symptoms such as tremor, bradykinesia, and rigidity are hallmark features, PD also includes cognitive and psychiatric manifestations3,4. The management of PD remains clinically challenging, due to limitations in diagnostic accuracy and the lack of tools for continuous symptoms monitoring. The average time needed for diagnosis in clinical setting can reach up to 2.9 years5. This diagnostic delay limits opportunities for early interventions, which is crucial for slowing disease progression, diminishing impact on daily activities, and identifying therapeutic targets and dosages6. However, the most pressing clinical need extends beyond initial diagnosis: continuous monitoring of symptom fluctuations and medication effects is essential for optimal patient care. Quality of life in PD is strongly correlated with a clinician’s ability to precisely titrate medications7, as patients experience fluctuating symptoms throughout the day and often develop motor complications, such as dyskenesia, with long-term dopaminergic therapy8. Current assessment via MDS-UPDRS9 provides only infrequent snapshots of the patient’s condition, through synchronized meetings with certified expert clinicians. Furthermore, both motor and non-motor symptoms require continuous monitoring: motor fluctuations often require medication adjustments, while non-motor symptoms, including cognitive changes and depression4 require early detection and management as they have an even greater impact on health-related quality of life than motor symptoms10. Speech analysis presents a promising avenue for PD for addressing these clinical challenges. Speech impairment, specifically hypokinetic dysarthria11, affects up to 90% of PD patients at some point during the disease course. Since speech production relies on precise neuromuscular control and complex cognitive linguistic processes12,13, it is an excellent candidate for PD diagnosis and monitoring. As speech production requires the coordinated and precise activation of articulatory and laryngeal muscles12, and cognitive planning and monitoring of multiple linguistic levels (phone, words, syntax, semantics, pragmatics)13, it is an excellent candidate for PD diagnosis and monitoring. Automatic speech analysis has shown potential beyond basic diagnosis, including differentiating L-dopa responsiveness14, distinguishing early and late onset PD15, assessing on-off medication states16, and estimating disease severity17. Specific features of speech, such as certain prosodic features18, disfluencies19, and prosodic prominence measures20 have been linked to cognition in PD. Despite these advances, current approaches have significant limitations. Most studies rely on single monolingual database, focus primarily on diagnosis, and do not evaluate systematically broader clinical requirements, such as tracking medication response, disease progression, and cognitive changes. Although recent work has shown that speech foundation models can outperform drastically classic speech features in PD detection21,22, many of these models lack reproducibility and multilingual cross-corpus evaluation. For instance, some models like Whisper cannot be independently replicated due to unavailable code and training details23. The multilingual analysis of PD introduces further complexity. Cross-linguistic variability presents challenges for developing universal models, and most research has been limited to handcrafted acoustic features, single tasks, or a narrow set of languages24–28. There is a pressing need for robust, cross-linguistic, replicable models that can generalize across diverse populations and settings without requiring large-scale data collection and annotation for each language. To address these clinical and methodological gaps, comprehensive evaluation frameworks must encompass multiple dimensions: cross-linguistic diagnosis (training on some languages and testing on others), medication state monitoring, disease severity staging, and systematic benchmarking across diverse linguistic populations. Such models are especially valuable due to global disparities in access to expert neurological care29,30. To address these challenges, we propose a self-supervised learning framework that leverages multiple multilingual databases to learn generalizable speech representations for PD assessment (Figure 1). Instead of relying on handcrafted features or language-specific models, our method learns robust representations that transfer effectively across languages and clinical tasks, enabling the comprehensive evaluation framework outlined above while maintaining reproducibility through publicly available code and clear methodological documentation. Overview of the ParkMAE pretraining and evaluation framework. (a) ParkMAE pretraining. We consolidated multiple large-scale databases spanning both general population and clinical datasets across four languages to pretrain our model. The input audio is transformed into a spectrogram, with a large portion masked. The model learns robust speech representations by reconstructing the masked regions using mean squared error (MSE) loss between reconstructed and original masked regions. This extensive multi-database pretraining enables cross-linguistic generalization. (b) In-domain evaluation. For each dataset separately, we extracted features using the pretrained frozen Speech Encoder and trained a linear classifier for various clinical tasks (diagnosis, medication state, disease staging). Results were averaged across five-fold cross-validation. (c) Cross-linguistic evaluation. We trained classifiers on consolidated data from three languages and evaluate on the fourth language, testing the model’s ability to generalize across linguistic and cultural boundaries without language-specific finetuning. The study by31 is the closest to our work among the existing literature on cross-language generalization. The authors compared interpretable features and non-interpretable features (extracted using deep learning models) across a collection of six datasets in various languages, in both in-domain (monolingual and multilingual) and cross-linguistic evaluation settings. The works of26,28 leveraged a wide variety of acoustic features to train their models on one of three languages and tested them on the remaining two languages, while27 included Taiwanese and Korean in the training set and then conducted evaluations separately on each language as well as on their combination. It is important to note that while numerous studies report success in detecting medication states16 and disease staging17 from speech, these achievements may not generalize across different datasets and populations. Many studies utilize carefully controlled clinical recordings or specific patient cohorts that may not reflect the heterogeneity found in broader clinical practice. Our work provides a systematic evaluation across multiple publicly available datasets to establish both the current capabilities and limitations of speech-based assessments, offering a comprehensive evaluation of what is practically achievable for PD management given current open source databases. Results We conducted two complementary experimental evaluations to assess the capabilities and limitations of speech-based PD assessment (Fig. 1): (A) In-domain evaluation, where models were trained and tested within each dataset separately using five-fold cross-validation, examining multiple clinical tasks including PD diagnosis, cognitive assessment (MoCA<26), off-medication detection, and disease staging (PDRS>12); and (B) Cross-linguistic evaluation, where models were trained on three languages and tested on a fourth to assess generalization across linguistic boundaries. Evaluation metrics We reported Positive Predictive Value (PPV), Negative Predictive Value (NPV), Sensitivity (Se), and Specificity (Sp)32,33 in the accompanying tables, with F1 scores visualized in the figures. The F1 score, defined as , provides a balanced measure of model performance particularly important for our imbalanced datasets. Trivial baseline results We chose a Naive Classifier as a trivial baseline, which always predicts the majority class. Since it yields F1-score 0.0 for the minority class, we included two additional trivial baselines in Table A3 in the Supplementary Information: the Always Positive Classifier (always predicts the positive class) and the Data Distribution Baseline (samples class labels randomly based on the underlying class distribution in the dataset). Note that the Naive baseline is referred to as the Majority Class baseline in Table A3. The Naive Classifier typically predicts the negative class, resulting in an F1-score and PPV of 0.0, except when the positive class is the majority (PDRS in PVA). The Always Positive Classifier achieves perfect sensitivity (1.00) but zero specificity (0.00), with moderate to high F1-scores when the positive class is balanced or frequent (PDRS in PVA). Finally, the Data Distribution Baseline performs between the other two, reflecting class proportions. All three trivial baselines perform at or near chance level (the AUC is around 0.5), with their strengths and weaknesses determined by class imbalance. In-domain performance In Fig. 2, we presented the in-domain evaluation results across all datasets and tasks. ParkMAE consistently achieved the highest performance across most tasks, with Whisper showing comparable results. Notable findings include: In-domain evaluation results. (A) F1 scores for each model across all tasks and datasets. Each bar represents test performance on the indicated language. (B) Detailed performance metrics (Sensitivity, Specificity, PPV, NPV) for ParkMAE. Note the near-chance performance on PVA dataset tasks, revealing limitations in medication state and disease staging detection. PD diagnosis: ParkMAE demonstrated robust diagnostic capability across all languages, achieving F1 scores ranging from 0.46 (Slovak EWA-DB34) to 0.98 (ItalianPVS35). The eGeMAPS36 baseline showed markedly inferior performance, particularly on the Slovak EWA-DB dataset where it failed entirely (F1 = 0.0), highlighting the limitations of traditional acoustic features for certain linguistic contexts. Cognitive assessment: for MoCA score prediction on the EWA-DB dataset, Whisper23 slightly outperformed ParkMAE (F1 = 0.58 vs 0.56). Both models showed moderate predictive capability above chance, suggesting speech captures meaningful cognitive information. Medication state and disease staging: critically, all models performed at or near chance levels for the PVA37 dataset tasks (PDRS severity and off-medication detection). The naive baseline, which simply predicts the majority class, achieved F1 scores of 0.86 and 0.0. Despite their sophistication, ParkMAE (0.84, 0.0) and Whisper (0.86, 0.0) did not show improvements. This reveals a fundamental limitation: current speech-based approaches cannot reliably distinguish disease stages or medication states using only the publicly available dataset PVA. Cross-linguistic generalization Figure 3 presents the cross-linguistic evaluation results for PD diagnosis—the only task consistently available across all four language datasets. These results reveal both the challenges of cross-linguistic transfer and the unique capabilities of ParkMAE. Cross-linguistic evaluation results for PD diagnosis. (A) F1 scores when training on three languages and testing on the fourth. Each bar represents test performance on the indicated language. (B) Detailed performance metrics (Sensitivity, Specificity, PPV, NPV) for ParkMAE. The consistent performance across diverse languages demonstrates the model’s ability to capture language-agnostic speech biomarkers of PD. All models experience substantial performance degradation when tested on unseen languages compared to in-domain evaluation, with F1 scores dropping by approximately 40-50% overall. This dramatic decrease highlights a fundamental challenge in developing universal speech biomarkers: models tend to overfit to language-specific acoustic patterns present in small clinical datasets, limiting their generalizability. Despite this overall performance drop, ParkMAE demonstrates remarkable resilience. With a mean cross-linguistic F1 score of , ParkMAE is the only model that consistently performs above chance across all test languages. In contrast, eGeMAPS catastrophically fails on multiple languages, achieving on Slovak and near-zero performance on Italian, rendering it clinically unsuitable for cross-linguistic applications. Whisper, despite its extensive multilingual training on 680,000 hours of speech data, shows inconsistent cross-linguistic performance (mean ), with several evaluation scenarios falling below clinically meaningful thresholds. The performance patterns reveal language-specific transfer difficulties. Slovak (EWA-DB) proves most challenging for all models, possibly due to its distinct phonological properties and the dataset’s acoustic characteristics. Conversely, Spanish (NeuroVoz) shows the highest cross-linguistic transfer success, with ParkMAE achieving , suggesting that some languages may share more disease-relevant acoustic features. Interestingly, our reconstruction-based pretraining appears to capture these language-agnostic disease markers more effectively than transcription-focused (Whisper) or handcrafted feature (eGeMAPS) approaches. The robustness of ParkMAE suggests that masked autoencoder pretraining successfully learns fundamental acoustic representations of neurological speech changes that generalize across linguistic differences. Model efficiency The performance and efficiency advantages of ParkMAE are particularly important: achieving optimal performance with 85M parameters represents an 89% reduction compared to Whisper-large (769M) while maintaining comparable accuracy. This suggests that targeted clinical pretraining with appropriate objectives can be more effective than scaling general-purpose models, with significant implications for practical deployment in clinical settings. Classifier and decision threshold tuning To ensure fair comparison across all models in our primary evaluation (Figs. 2 and 3), we used Logistic Regression with a fixed threshold of 0.5. We subsequently explored whether performance could be improved through classifier optimization and threshold tuning using the TunedThresholdClassifierCV method from scikit-learn38. Tables A1 and A2 in the Supplementary Information present the best-performing configuration for each model (eGeMAPS, Whisper, and ParkMAE) in both in-domain and cross-linguistic settings. Note that cross-linguistic evaluation was limited to PD diagnosis, as secondary clinical measures (MoCA, PDRS, medication status) are not consistently available across all language datasets. Classifier optimization and threshold tuning substantially improved performance across all models, with higher F1-scores and AUCs. XGBoost generally performed best among the tested classifiers. However, certain tasks (PDRS, off-medication) remained challenging regardless of optimization. Importantly, ParkMAE maintained its superiority: it achieved the highest F1-scores and AUCs on nearly all tasks, with the highest average F1 across both in-domain and cross-linguistic settings. Notably, the AUC scores increased by around 25% in the cross-linguistic evaluation (except for MDVR). Threshold tuning also improved sensitivity for all datasets. For PD diagnosis, sensitivity was already high except in EWA-DB, where it nearly doubled (while specificity remained high). Discussions Our comprehensive evaluation of speech-based Parkinson’s disease assessment reveals both significant advances and important limitations in the field of speech assessment of PD. ParkMAE outperformed eGeMAPS and demonstrated comparable performance to Whisper-large while using a fraction of the computational resources. However, our results also exposed critical gaps for practical capabilities for medication monitoring and disease staging. Clinical capabilities of speech modeling The contrast between diagnostic success and monitoring of disease severity and medication failures represents an important finding of our study. ParkMAE achieves diagnostic F1 scores ranging from 0.46 to 0.98 across different languages, demonstrating that speech-based PD detection is mature enough for clinical deployment, consistent with recent advances using foundation models21,22. However, our inability to reliably detect medication states or disease stages on the PVA37 dataset—with performance barely exceeding chance–requires careful interpretation. We refer to the Naive Classifier, which deterministically predicts the majority class, as chance. As discussed in “Trivial baseline results”, Table A3 in the Supplementary Information shows the results for all three trivial baseline results. All trivial baselines performed around chance level (AUC 0.5) while ParkMAE achieved predictive capability above chance, with AUC values ranging from 0.74 to 1.0, except for the PVA dataset. Although sensitivity for ItalianPVS, NeuroVoz, and MDVR approaches that of the Always Positive classifier, there remains room for improvement in sensitivity for EWA-DB, which may be due to class imbalance. As discussed in “Classifier and decision threshold tuning”, sensitivity can be significantly improved for EWA-DB (it nearly doubled) with careful classifier selection and threshold tuning. The Data Distribution baseline offers a more class-balanced performance (compared to the other two baselines), and we can observe that ParkMAE significantly outperforms this baseline. After classifier and threshold tuning in Tables A1 and A2 in the Supplementary Information, ParkMAE still outperforms eGeMAPS, Whisper and the trivial baselines. The higher AUC scores indicate that a model can better distinguish between positive and negative cases, leading to more accurate diagnoses. Besides the AUC scores, sensitivity also improved (as discussed in “Classifier and decision threshold tuning”, it nearly doubled for EWA-DB). High sensitivity is crucial in clinical applications to minimize missed diagnoses. The best AUCs exceeded 0.9 on ItalianPVS, NeuroVoz, and MDVR, confirming that our self-supervised pretraining learns robust representations that outperform both handcrafted features and general-purpose speech models even when all approaches are optimally configured. Two explanations emerge for the medication monitoring failure. First, speech may genuinely be insensitive to short-term medication changes, as supported by39, who found no speech alterations from off to on states in any investigated speech dimensions (monopitch, monoloudness, imprecise consonants, harsh voice, slow sequential motion rates, articulation rate, or inappropriate silences) despite obvious motor improvements. They concluded that “speech assessment may provide a medication state-independent motor biomarker of PD”, aligning perfectly with our findings. Alternatively, the PVA dataset’s limitations–containing only PD patients performing a single sustained vowel task–may be insufficient to capture medication-related changes that other studies have reported14,16. The dataset’s homogeneity and restricted task may mask subtle variations that richer speech samples could reveal. For disease staging, the failure across all models suggests similar constraints. While17 reported success in severity assessment, their approach used comprehensive speech tasks and carefully controlled recordings. The limited scope of PVA dataset may explain our inability to replicate these findings, highlighting how dataset quality fundamentally constrains model capabilities regardless of architectural sophistication. The moderate success in cognitive assessment offers a more encouraging perspective. Our models achieved F1 scores of 0.53 for MoCA threshold detection on the EWA-DB dataset, demonstrating predictive capability above chance. This aligns with growing evidence linking speech markers to cognitive function in PD. In18, authors found that speech prosody impairment predicted cognitive decline, while20 demonstrated correlations between prosodic prominence measures and cognitive abilities. Recent work showed that connected speech tasks and specific disfluencies are sensitive to cognitive changes in neurodegenerative diseases19, suggesting that open vocabulary (OV) and constrained vocabulary (CV) tasks in EWA-DB may better capture cognitive-linguistic processing than simple phonation tasks. These differential results across clinical dimensions suggest that speech modeling may be inherently better suited for certain assessments. The robust diagnostic performance across all datasets indicates that the gross neurological changes of PD consistently manifest in speech. The promising cognitive results suggest speech captures higher-level linguistic processing affected by PD’s non-motor symptoms. However, the failure to detect medication states or motor severity staging–despite literature claims–indicates either fundamental limitations of speech biomarkers for these tasks or the need for more sophisticated data collection protocols than currently available in public datasets. Cross-linguistic generalization for a wider deployment The 40–50% performance drop in cross-linguistic evaluation revealed the challenge of developing universal speech models. This degradation observed across all models indicates systematic overfitting to language or specific patterns in small clinical datasets, a challenge also recognized in other recent cross-linguistic studies25,40. Despite the non-consistent performance in cross-linguistic setting, ParkMAE’s ability to generalize effectively to unseen languages without additional finetuning represents meaningful progress. This suggests that the model learns language-independent speech markers of Parkinson’s disease, making it promising for use across different countries and populations. The failure of eGeMAPS on certain languages (F1 = 0.06 for Slovak) demonstrates the inadequacy of handcrafted features for cross-linguistic applications—despite their theoretical language independence36—and suggests a potential bias toward languages that are better represented in the speech community. Even Whisper, trained on 680,000 h of multilingual data23, shows erratic cross-linguistic performance, suggesting that optimizing for speech recognition may not align with learning disease-relevant acoustic patterns. We observed that cross-linguistic results depend considerably depending on the test language. While the results for Spanish and English are very promising, testing on Slovak proved to be particularly challenging. Multiple hypotheses may explain this disparity and the varying results. First, deep learning architectures are predominantly optimized for English. Although both Whisper and ParkMAE were pretrained on multiple languages, the majority of their pretraining data is in English, reflecting the predominance of English in available datasets. While clinical datasets exist in various languages, they are often limited in size. Indeed, the largest pretraining datasets used for ParkMAE–SAP (304 h) and mPower (176 h)—are in English, while the other clinical datasets are considerably smaller, each containing less than 10 h of data. These observations highlight the need for richer and more diverse datasets with a wide range of linguistic, demographic, and clinical speech data to overcome current limitations in cross-linguistic clinical assessment. Second, differences in audio tasks may also affect the evaluation results, as the datasets contain a diverse set of audio tasks that are not consistent across datasets. For example, NeuroVoz includes the greatest variety of audio tasks (30 in total), whereas MDVR provides only one text reading and one spontaneous dialogue task per speaker. Even within a dataset, not all patients necessarily completed every audio task or the same set of tasks. Last but not least, the datasets vary in size and were collected under a range of environments and with different types of recording equipment (clinical-grade microphones versus mobile devices in MDVR41). In addition, there are differences in class balance across the datasets. While ItalianPVS, MDVR, and NeuroVoz are relatively balanced, EWA-DB contains a much larger proportion of control patients compared to those with Parkinson’s disease. Besides, the stages of Parkinson’s disease may not be uniformly represented in the datasets, which may further affect model performance. Recent studies support the feasibility of cross-linguistic PD detection26–28,31,40,42. In42, authors demonstrated that certain acoustic features of PD speech remain consistent across languages, while31 achieved successful cross-language detection using deep learning approaches and identified language-independent biomarkers across multiple tasks. In the cross-linguistic experiments, they also concluded that the performance of the evaluated models varied significantly depending on the test language for both interpretable and non-interpretable features. Besides, non-interpretable features also achieved superior results compared to interpretable ones. The variation in cross-linguistic transfer—with Slovak proving most challenging and Spanish easiest—offers insights into the universality of PD speech modeling. This also aligns with24, who found consistent speech biomarkers across Czech, German, French, and American English speakers, suggesting that fundamental neurological changes transcend linguistic boundaries while surface-level acoustic features may vary. The work of26 studied the effect of incrementally including a portion of the test language in the training dataset, ranging from 0% up to 80% in steps of 10%. They concluded that the results improved rapidly, demonstrating that including even a small portion (e.g., 20%) of the test language in the train data significantly improves the results. Furthermore43, proposed federated learning to leverage multilingual data while preserving privacy. Our results extend these findings by demonstrating that self-supervised pretraining on acoustic reconstruction can capture these universal patterns more effectively than supervised or transcription-based approaches. Model efficiency and reproducibility Beyond performance metrics, ParkMAE’s efficiency addresses practical deployment barriers which is of utmost importance given the PD epidemiology data30. Figure 4 presents the model efficiency analysis. Achieving superior results with 85M parameters—88.8% fewer than Whisper-large–while maintaining transparent training procedures enables deployment scenarios, with only open source datasets, previously impractical for resource-constrained healthcare settings. Model efficiency analysis. F1 score versus model size (millions of parameters) for cross-linguistic PD diagnosis. ParkMAE (85M parameters) achieves comparable or superior performance to larger Whisper models, demonstrating that effective cross-linguistic generalization does not require massive model scale. The contrast with Whisper exemplifies broader concerns about black-box models in healthcare. As44 indicates, future OpenAI audio models will not be systematically open-sourced, limiting scientific reproducibility and clinical adaptation. Besides, there is no training code or dataset shared publicly for the Whisper family models. For medical applications requiring regulatory approval, the ability to audit training data and code and replicate results is essential45. Our complete documentation and use of open source toolkit only enables independent validation, which is critical given recent concerns about bias and fairness in medical AI46. Limitations While the results of ParkMAE are promising, this study has some limitations. Our evaluation datasets were collected under varying protocols across different institutions and countries. While all diagnoses were made by qualified neurologists, differences in recording equipment (e.g., clinical-grade microphones vs. mobile devices41), environments (clinical vs. home settings), and task diversity may contribute to the observed performance variations across datasets. However, this heterogeneity also validates our model’s ability to generalize to real-world clinical scenarios where such variations are inevitable. While it would be clinically meaningful to distinguish between different neurodegenerative disorders, our evaluation was limited in this regard. EWA-DB is the only evaluation dataset that includes patients with a neurodegenerative disease other than Parkinson’s disease, specifically Alzheimer’s disease. However, we did not include these patients in the evaluation procedure to maintain consistency with the other evaluation datasets. Addressing class imbalance and exploring multiclass approaches represent important directions for future research. Moreover, it would be valuable to investigate the use of standardized tasks to elicit features that distinguish between different neurodegenerative diseases. Methods Datasets We leveraged a collection of large-scale pretraining datasets and clinically relevant evaluation datasets to train and test our models. There is no overlap between the pretraining datasets and evaluation datasets. The pretraining datasets consist of both general population speech corpora and datasets specifically collected from individuals diagnosed with neurodegenerative and psychiatric diseases. The evaluation datasets were chosen not only to assess PD detection performance but also to evaluate the model’s generalization across different languages, speech tasks, and clinical assessments, including cognitive function (MoCA) and disease severity measures such as the Parkinson’s Disease Rating Scale (PDRS). By combining large-scale pretraining datasets and clinically rich evaluation datasets, we designed a robust experimental framework to develop speech-based biomarkers for Parkinson’s disease. The pretraining datasets provide a strong foundation by covering diverse speech patterns and speaker demographics, while the evaluation datasets ensure that the model is tested across different languages, voice tasks, and disease assessments. These datasets cover a variety of speech tasks that provide complementary information about articulation, fluency, and voice quality. Constrained Vocabulary (CV) tasks involve reading predefined texts, such as counting numbers or repeating standard phrases, ensuring controlled speech conditions. Open Vocabulary (OV) tasks require participants to engage in spontaneous speech, such as answering open-ended questions, capturing more natural linguistic and prosodic variations. Vowel (V) tasks assess voice quality through sustained phonation of a single vowel sound (e.g., “a”), which is useful for detecting vocal tremors and breathiness. Finally, Diadochokinetic (DDK) tasks measure speech motor control and articulation speed by having participants rapidly repeat syllables (e.g., “pa-ta-ka”). Pretraining datasets The pretraining datasets include a mix of large-scale speech corpora designed for speaker recognition, general speech analysis, and clinical research on neurodegenerative diseases. Table 1 provides an overview of these datasets. By training on a combination of general population and clinical datasets, our model learned robust speech representations while incorporating disease-specific speech features. Table 1Overview of pretraining datasets. This table summarizes the large-scale speech datasets used for model pretraining. The datasets include both general population corpora and clinically focused databases. Clinical datasets comprise healthy controls as well as patients with Parkinson’s disease (PD), amyotrophic lateral sclerosis (ALS), cerebral palsy (CP), down syndrome (DS), and stroke. The variety of speech tasks enables learning robust representations of speech, while clinical datasets contribute to disease-specific features (CN: Chinese, EN: English, IT: Italian).Dataset#Speakers#Pathological/control#AudiosSpeech hoursClinical relevanceClinicalAndroids corpus (IT)11058/522207h33DepressionMODMA (CN)5223/2915037h11DepressionmPower (EN)5038970/3986/8263,208176h14PDSAP (EN)424424/0135,668304h21PD, ALS, CP, DS, strokeTorgo (EN)158/713201h13ALS, CPVOC-ALS (IT)153102/5112245h00ALSGeneralCLAC (EN)1832–18,609104h45–40% VoxCeleb1 (EN)1187–58,478140h38–Total8811280,230746h55In mPower49, there are 970 PD patients, 3986 control patients and 82 patients with unknown diagnostic. Overview of pretraining datasets. This table summarizes the large-scale speech datasets used for model pretraining. The datasets include both general population corpora and clinically focused databases. Clinical datasets comprise healthy controls as well as patients with Parkinson’s disease (PD), amyotrophic lateral sclerosis (ALS), cerebral palsy (CP), down syndrome (DS), and stroke. The variety of speech tasks enables learning robust representations of speech, while clinical datasets contribute to disease-specific features (CN: Chinese, EN: English, IT: Italian). In mPower49, there are 970 PD patients, 3986 control patients and 82 patients with unknown diagnostic. Androids corpus47. The Androids Corpus is an Italian dataset designed for automatic depression detection. It collects CV and OV speech recordings from both depressed patients and healthy controls, with diagnoses made by professional psychiatrists. MODMA48. MODMA is a Chinese dataset that consists of audio recordings of clinically depressed patients and healthy controls, aimed at supporting research on mental disorder analysis. The CV and OV audio tasks were associated with three types of emotional valences: positive, neutral, and negative. mPower49. The mPower dataset is a large-scale dataset that collects speech samples from English-speaking patients with Parkinson’s disease and healthy controls from the United States. The dataset focuses on sustained phonation of the vowel “a”, which is commonly used in clinical voice assessments. SAP50. This clinically curated dataset comprises diverse speech recordings from English-speaking individuals in the United States with diagnosed speech impairments, including Parkinson’s disease, amyotrophic lateral sclerosis (ALS), cerebral palsy, down syndrome, and stroke survivors. Torgo51. This Canadian English database collects CV, DKK, OV and V speech tasks from individuals with dysarthria due to cerebral palsy or ALS as well as from healthy controls. TORGO was created to support the development of Automatic Speech Recognition (ASR) models that are more suited for the communication needs of people affected by dysarthria. VOC-ALS52. VOC-ALS consists of Italian-speaking ALS patients and healthy controls. The ALS patients exhibit varying degrees of dysarthria, and the speech tasks include DKK and V tasks. CLAC53. CLAC is a large English-speaking general population dataset with participants primarily from the United States. It includes a variety of CV and OV speech tasks. It contributes to training models on everyday speech variations. VoxCeleb154. VoxCeleb1 is a widely used speaker identification dataset consisting primarily of English speech samples extracted from YouTube video interviews. We selected 40% of VoxCeleb1 to maintain balanced representation across dataset types in our pretraining corpus: 70% clinical PD data (mPower and SAP), 20% general population data (CLAC and VoxCeleb1), and 10% other neurological and psychiatric conditions (Androids Corpus, MODMA, Torgo, and VOC-ALS). Using less VoxCeleb1 or switching to VoxCeleb2 would have disrupted this carefully designed balance. Evaluation datasets To assess the performance of the models in real-world clinical settings, we selected five clinical speech datasets containing recordings from both PD patients and healthy controls. These datasets were chosen for their diversity in language, voice tasks, and clinical assessments, as detailed in Table 2. Note that these datasets are different from the pretraining datasets. Table 2Overview of evaluation datasets. This table provides a summary of the clinically relevant datasets used for model evaluation, detailing the types of speech tasks recorded, number of speakers and recordings, total speech duration, and clinical assessments available. The datasets were selected to ensure diversity in language, speech tasks, and clinical assessments, including PD diagnosis and severity scales such as MoCA. The distribution of PD-positive (+) and control (−) speakers is also provided. (EN: English, ES: Spanish, IT: Italian, SK: Slovak).Dataset#Speakers#PD/control#AudiosSpeech hoursClinical assessmentEWA-DB (SK)95695/861669288h10PD diagnosis + MoCAItalianPVS (IT)5927/327354h57PD diagnosisMDVR (EN)3615/21722h50PD diagnosisNeuroVoz (ES)10550/552,8213h40PD diagnosisPVA (EN)620620/07793h35PDRS + On/off medication Overview of evaluation datasets. This table provides a summary of the clinically relevant datasets used for model evaluation, detailing the types of speech tasks recorded, number of speakers and recordings, total speech duration, and clinical assessments available. The datasets were selected to ensure diversity in language, speech tasks, and clinical assessments, including PD diagnosis and severity scales such as MoCA. The distribution of PD-positive (+) and control (−) speakers is also provided. (EN: English, ES: Spanish, IT: Italian, SK: Slovak). EWA-DB34. The EWA-DB dataset is one of the largest PD speech datasets, containing Slovak PD and healthy control individuals. Participants completed a variety of speech tasks, including all types of speech tasks (CV, DDK, OV and V). Additionally, this dataset includes Montreal Cognitive Assessment (MoCA) scores, which allows the model to predict cognitive decline in addition to PD diagnosis. Although EWA-DB contains CV tasks (reading pre-defined nouns and verbs), we decided to omit these from the evaluation due to the large size of the dataset. Moreover, we did not include patients with Alzheimer’s disase. ItalianPVS35. ItalianPVS (Italian Parkinson’s Voice and Speech dataset) is an Italian PD dataset. It focuses on CV, DKK, and V tasks and is specifically designed for assessing speech intelligibility in PD patients. MDVR41. The MDVR (Mobile Device Voice Recordings) dataset contains mobile-recorded speech from UK-based participants. It includes CV and OV tasks, allowing the model to generalize across structured and spontaneous speech. NeuroVoz55. This dataset is a Castilian Spanish dataset. It includes all speech tasks (CV, DDK, OV and V) and Unified Parkinson’s Disease Rating Scale (UPDRS) scores, which are crucial for assessing PD severity beyond simple diagnosis. PVA37. The PVA (Patient Voice Analysis) dataset is composed exclusively of PD patients, collected to analyze disease severity. Recordings focus on sustained phonation of the vowel “a”. Clinical scores include Hoehn and Yahr (HY) scale and Parkinson’s Disease Rating Scale (PDRS) assessments. Additionally, it provides on/off medication labels, allowing models to explore the effects of PD medication on speech. Speech encoder models Baseline models We compared the performance of ParkMAE with the following baseline models. Naive classifier: the naive classifier is a simple and deterministic classifier that predicts the majority class. If a model underperforms this baseline during evaluation, it indicates that the model is not learning. eGeMAPS36: eGeMAPS is a comprehensive acoustic speech feature set. It is widely used for speech analysis due to its interpretability and language independence. Whisper23: whisper is an automatic speech recognition model that was trained on nearly 700,000 hours of multilingual data to predict audio transcripts. Whisper leverages a Transformer encoder-decoder architecture56. The encoder part of the model is particularly effective at extracting rich audio representations. Masked autoencoders Masked autoencoder (MAE) models were introduced in57 as self-supervised models for computer vision. Self-supervised pretraining offers several advantages. It enables models to learn meaningful representations from unlabeled datasets, making it especially valuable in domains where labeled data is scarce or expensive to obtain, such as in the medical field. Self-supervised methods, including the Masked Autoencoders approach57, benefit from large and diverse pretraining datasets58. It can be pretrained on a collection of datasets spanning multiple languages and diverse domains, including neurodegenerative and psychiatric diseases as well as general population data. The core idea of MAEs is to divide an image into non-overlapping patches, mask a large portion of them randomly, and reconstruct the original image with an encoder-decoder architecture. The non-masked patches are passed through a Vision Transformer (ViT)58 encoder to extract latent representations. Then a lightweight ViT decoder processes the full set of patches, both masked and non-masked, to reconstruct the original image. Mean squared error (MSE) loss is applied between the reconstructed masked patches and the original masked patches. The use of MAE was extended to audio processing in AudioMAE59, by applying an MAE to spectrograms. In OPERA60, the authors pretrain an audio MAE on a collection of respiratory datasets. We pretrained our MAE model, called ParkMAE, on a collection of datasets presented in Table 1, comprising of both general population and clinical speech recordings. The encoder is a ViT-B model with 85M parameters and the decoder is a Swin Transformer61 model with 51M parameters. The input audios (mono channel with sampling rate 16,000 Hz) are 10 second audio slices: longer audios were randomly cropped and shorter ones were padded with 0. Then they were transformed into 128 Mel-frequency bands with a 25 ms Hanning window and 10 ms window shift to obtain spectrograms of size 1 1024 128. We normalized the input spectrograms to the range [0, 1] before feeding them to the model backbone. The model was initialized with the pretrained AudioMAE model weights. We used patch size 16 16, Adam optimizer, initial learning rate 0.0001, batch size 256, and pretrained the model for 100 epochs. Model evaluation Clinical tasks As illustrated in Fig. 1b-c, the pretrained models were evaluated using two approaches. The binary classification tasks came from the clinical assessment available in each corpus, provided in Table 2. For PD diagnosis, we predicted the presence of Parkinson’s disease (positive class), and for the on/off medication assessment, we focused on identifying the off-medication state. Furthermore, we defined thresholds for the MoCA and PDRS clinical assessment scores, creating the following binary classification tasks: MoCA < 26 to identify cognitive impairment, and PDRS > 12 to identify disability in Parkinson’s disease, respectively. Note that the PDRS score in the PVA dataset is a shortened version of the UPDRS scale, and is defined as follows. The PDRS questionnaire consists of 17 questions, each scored on a scale from 0 to 4, where “0” indicates no disability. The overall PDRS score is calculated as the floor of the average score across the 17 items, rescaled from 0–4 to 0–100 by multiplying by 25. Since “0” represents no disability and “1” represents minimal disability, we set a threshold of 0.5 for each question, which yields a 12 threshold on the total score for the rescaled PDRS scale. This threshold was chosen to reflect the clinical manifestation of Parkinson’s disease symptoms. Note that every patient in PVA has Parkinson’s disease. Linear probing The models are evaluated with the linear probing approach. Linear probing consists of training a linear classifier on top of the features of a pretrained Speech Encoder. This technique provides an efficient way to assess how effectively the learnt representations can be transferred to new tasks and new datasets, without the need to retrain the entire Speech Encoder, which would be computationally expensive. The audio recordings are split into overlapping windows, with the overlap being half the size of the window. The window size is 10 seconds for ParkMAE, while it is 30 seconds for eGeMAPS and Whisper. For each slice of an audio, a feature vector is computed by a Speech Encoder. Ground truth binary labels are assigned to each slice corresponding to the ground truth binary label of the full audio. A Logistic Regression classifier is then trained on the collection of all feature vectors in the dataset, treating the audio slices independently. During the evaluation phase, a feature vector is extracted for each window slice of the input audio with the pretrained model, and then a prediction is made by the classifier. The predictions are averaged with mean pooling to predict a probability. Finally, these probabilities are rounded to a binary label based on a threshold value. To ensure fair comparison across all models (eGeMAPS, Whisper, and ParkMAE), we consistently used Logistic Regression with a fixed decision threshold of 0.5 throughout the main evaluation. This standardized approach prevents confounding effects from classifier choice and allows direct assessment of the learned representations’ quality. Additionally, we conducted supplementary experiments with optimized classifiers (including XGBoost, SVM, and Random Forest) and threshold tuning to demonstrate achievable performance bounds for clinical deployment. The best results are reported in Tables A1 and A2 in the Supplementary Information for all models (eGeMAPS, Whisper-large, and ParkMAE). In-domain linear probing: for each evaluation dataset separately, we evaluated the models within the same dataset. Due to the limited size of the evaluation datasets, we applied five-fold cross-validation and reported the mean results averaged across the five folds. Cross-linguistic linear probing: while we evaluated the models on all five evaluation datasets for in-domain linear probing, we used only four of these datasets for cross-linguistic linear probing: EWA-DB, ItalianPVS, MDVR and NeuroVoz. Note that these four datasets represent four different languages. We successively used a combination of three out of the four datasets as the training set, and the remaining dataset as the test set. Note that the test dataset is in a language that is not present in the train set. For cross-linguistic linear probing, we focused solely on binary classification of PD diagnosis, as this task is consistently available across all included datasets. PVA is excluded since it contains only PD patients and no healthy controls, resulting in a single class for the diagnosis label. To ensure a rigorous cross-linguistic evaluation procedure, we required only one English dataset, MDVR in this case. Although PVA includes English-speaking patients, only the sustained phonation of the vowel “a” is available, whereas MDVR provides OV and CV tasks, making it more suitable for our analysis. Conclusion This study introduces ParkMAE, a self-supervised multilingual speech foundation model and benchmarking system designed for Parkinson’s disease assessment. The model was pretrained on approximately 750 hours of multilingual corpus and diverse clinical populations including Parkinson’s disease, general population and additional psychiatric and neurological conditions. Leveraging a self-supervised pipeline, ParkMAE is able to extract meaningful features from large amounts of unlabeled and diverse data, enabling robust assessment within a flexible framework. ParkMAE demonstrated strong diagnostic performance for Parkinson’s disease in an extensive evaluation procedure, achieving robust results across both in-domain and cross-linguistic settings, while using significantly fewer model parameters than Whisper-large and providing a reproducible framework for future clinical development. Cross-linguistic evaluation proved to be more challenging than the in-domain setting, highlighting the challenge of building universal speech models for Parkinson’s disease. The poor performance of handcrafted features and even large multilingual models like Whisper emphasizes the difficulty of cross-linguistic transfer. Cross-linguistic evaluation results varied considerably depending on the test language. This disparity may be due to several factors, such as deep learning models being primarily optimized for English, as most pretraining and clinical datasets are English-dominant. Furthermore, differences in audio tasks, dataset sizes, and recording environments and equipments likely contribute to these variations. However, our findings also reveal some limitations: all models experience performance drops in the cross-linguistic setting. Moreover, further work is needed to address class imbalance, extend models to multiclass classification, and distinguish among different neurodegenerative disorders. Exploring standardized tasks to identify features that differentiate these disorders would be valuable for clinical research. Despite the performance drops in the cross-linguistic setting, ParkMAE showed the ability to generalize to unseen languages without language-specific finetuning. This suggests that the model captures fundamental speech features associated with Parkinson’s disease that are consistent across linguistic boundaries, making it suitable for deployment in diverse clinical settings and populations. Besides, the results of ParkMAE demonstrated that speech-based models could offer valuable insights into cognitive function. Although the current performance leaves room for improvement, the model’s predictive capability highlights the potential for non-invasive, speech-based cognitive monitoring in clinical practice. Supplementary Information Supplementary Information. Acknowledgements This project was provided with computing AI and storage resources by GENCI at IDRIS thanks to the grant 20XX-AD010315777 on the supercomputer Jean Zay’s V100/A100 partition. Data availability All pretraining and evaluation datasets are either publicly available or can be obtained upon request from the corresponding authors to ensure reproducibility. Code availability Our codebase for the ParkMAE training pipeline is based on the AudioMAE59 and Opera60 repositories, both of which are publicly available on the following links: https://github.com/facebookresearch/AudioMAE and https://github.com/evelyn0414/OPERA, respectively. Declarations Competing Interests All authors have shares in Callyope. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Angelika Ando, Adrien Lesage and Marc De Gennes contributed equally to this work. Supplementary Information The online version contains supplementary material available at 10.1038/s41598-025-30251-7. References Associated Data Supplementary Materials Supplementary Information. Data Availability Statement All pretraining and evaluation datasets are either publicly available or can be obtained upon request from the corresponding authors to ensure reproducibility. Our codebase for the ParkMAE training pipeline is based on the AudioMAE59 and Opera60 repositories, both of which are publicly available on the following links: https://github.com/facebookresearch/AudioMAE and https://github.com/evelyn0414/OPERA, respectively.