HPP-Voice: A Large-Scale Evaluation of Speech Embeddings for Multi-Phenotypic Classification

David Krongauz, Hido Pinto, Sarah Kohn, Yanir Marmor, Eran Segal · 2025

Read HPP-Voice: A Large-Scale Evaluation of Speech Embeddings for Multi-Phenotypic Classification by David Krongauz, Hido Pinto, Sarah Kohn, Yanir Marmor, Eran Segal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16490.txt.

Introduction

David Krongauz and co-authors test whether thirty seconds of counting aloud can betray a person's health, in work from 2025. Voices carry cues about body and brain states, raising hopes for cheap, non-invasive screening.

The HPP-Voice corpus holds 7,188 recordings of Hebrew-speaking adults, each linked to as many as 15 health traits. Fourteen modern speech embedding models were compared against classic features and demographics. Embeddings won broadly. A speaker-identification model best predicted serious sleep apnea in men. Patterns differed by sex: identity and diarization models led for men's breathing and sleep traits, while smoking and anxiety in women favored other model families. The results map which voice models suit which conditions.

Catalog record

CREATORDavid Krongauz, Hido Pinto, Sarah Kohn, Yanir Marmor, Eran Segal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16490

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HPP-Voice: A Large-Scale Evaluation of Speech Embeddings for Multi-Phenotypic Classification David Krongauz, Hido Pinto, Sarah Kohn, Yanir Marmor, Eran Segal Abstract Human speech contains paralinguistic cues that reflect a speaker’s physiological and neurological state, potentially enabling non-invasive detection of various medical phenotypes. We introduce the Human Phenotype Project Voice corpus (HPP-Voice): a dataset of 7,188 recordings in which Hebrew-speaking adults count for 30 seconds, with each speaker linked to up to 15 potentially voice-related phenotypes spanning respiratory, sleep, mental health, metabolic, immune, and neurological conditions. We present a systematic comparison of 14 modern speech embedding models, where modern speech embeddings from these 30-second counting tasks outperform MFCCs and demographics for downstream health condition classifications. We found that embedding learned from a speaker identification model can predict objectively measured moderate to severe sleep apnea in males with an AUC of 0.64 ± 0.03, while MFCC and demographic features led to AUCs of 0.56 ± 0.02 and 0.57 ± 0.02, respectively. Additionally, our results reveal gender-specific patterns in model effectiveness across different medical domains. For males, speaker identification and diarization models consistently outperformed speech foundation models for respiratory conditions (e.g., asthma: 0.61 ± 0.03 vs. 0.56 ± 0.02) and sleep-related conditions (insomnia: 0.65 ± 0.04 vs 0.59 ± 0.05). For females, speaker diarization models performed best for smoking status (0.61 ± 0.02 vs. 0.55 ± 0.02), while Hebrew-specific models performed best (0.59 ± 0.02 vs. 0.58 ± 0.02) in classifying anxiety compared to speech foundation models. Our findings provide evidence that a simple counting task can support large-scale, multi-phenotypic voice screening and highlight which embedding families generalize best to specific conditions, insights that can guide future vocal biomarker research and clinical deployment. 1 Introduction Human speech is a richly layered signal. Beyond lexical content, it carries prosody, articulation, timing, respiration, and other paralinguistic cues that mirror a speaker’s physiology and neurological state [ 11 , 12 ] . Recent deep learning work has capitalized on these cues for emotion recognition [ 23 ] , speaker verification [ 28 ] , and detection of neurodegenerative [ 44 ] or pulmonary disorders [ 38 ] . However, most clinical studies remain narrow: they target a single disease (e.g., Parkinson’s disease [ 30 ] and Alzheimer’s disease [ 25 ] ), use small cohorts, or focus on non-speech events such as coughs and breaths [ 4 ] . In this study, we leverage the extensive and uniquely comprehensive data from the Human Phenotype Project (HPP) [ 39 ] to investigate associations between voice characteristics and health conditions. Furthermore, we systematically compare multiple voice feature extraction techniques, ranging from classical spectral features to advanced embeddings 1 1 1 The terms embeddings and representations are used interchangeably throughout this paper. derived from state-of-the-art self-supervised foundation models for speech. Why fluent counting? A brief “count-to-30” prompt is easy to administer remotely, highly reproducible, and intuitively understood by speakers of any language [ 49 , 37 ] . It also elicits sustained phonation and prosodic variability, which are not captured in cough-centric corpora such as Coswara [ 38 ] . The HPP-Voice corpus. We introduce HPP-Voice , comprising 7,188 recordings from 6,760 adults (3,211 males / 3,549 females, age 52\pm 10 years). Each utterance is paired with 15 health conditions spanning across 6 body systems: respiratory, metabolic, neurological, mental health, immune, and sleep [ 39 ] . Our contributions are: 1. Large, clinically diverse corpus. We release a novel dataset of fluent speech paired with verified diagnoses across six body systems. 2. Comprehensive benchmark. Fifteen state-of-the-art encoders (e.g., MFCC [ 10 ] , x-vector [ 40 ] , wav2vec 2.0 [ 3 ] , WavLM [ 8 ] ) are evaluated under identical splits. 3. Phenotype-specific insights. We demonstrate which speech representations outperform for specific clinical domains, providing a comprehensive evaluation of their relative efficacy across different healthcare applications. 2 Related Work Early work on voice-based clinical biomarkers focused primarily on single conditions. These studies relied on handcrafted features related to prosody and phonation to monitor diseases such as Parkinson’s [ 29 , 30 ] , amyotrophic lateral sclerosis (ALS) [ 41 ] , Alzheimer’s disease [ 25 ] , and various laryngeal disorders [ 19 , 34 ] . However, such efforts were often constrained by small sample sizes (typically dozens to a few hundred participants), limiting statistical power and the ability to derive insights across conditions or populations. Recent efforts to scale clinical voice datasets have largely focused on non-speech vocalizations. The COVID-19 pandemic accelerated the creation of cough- and breath-centric corpora, such as Coswara [ 38 ] , MIT-Cough [ 22 ] , and the HeAR dataset [ 4 ] , which aggregates recordings from Coswara, CoughVID [ 32 ] , and CIDRZ. Although these datasets include thousands of audio samples, they primarily capture explosive airflow events rather than sustained phonation or prosodic variation, thus limiting their utility for analyzing speech patterns. In parallel, advances in self-supervised learning (SSL) have led to powerful speech representations trained without explicit labels. Models such as Wav2Vec 2.0 [ 3 ] and WavLM [ 8 ] now dominate performance benchmarks across a range of tasks [ 51 ] , including emotion recognition [ 48 , 23 ] and speaker identification [ 16 ] . SSL embeddings consistently outperform traditional baselines such as MFCCs and x-vectors, but prior work has not yet compared such a diverse set of models on a single, clinically annotated speech corpus [ 4 ] . Efforts to develop voice-based screening for multiple conditions remain rare, with notable exceptions including COPD-Gene Voice [ 15 ] and HeAR [ 4 ] , which focus primarily on respiratory traits. We fill this gap by introducing HPP-Voice and conducting the first head-to-head comparison of fifteen modern encoders on a large, clinically annotated speech corpus for multi-morbid screening. 3 Dataset: HPP-Voice HPP-Voice is the voice arm of the Human Phenotype Project (HPP), a longitudinal, multi-omics registry designed to unravel chronic-disease mechanisms at the population scale. Cohort description The HPP cohort currently comprises 11,460 adults. Between December 2019 and December 2024, 6,760 unique speakers were invited for voice sampling and completed the protocol, yielding 7,188 recordings. Among the 6,760 speakers, 3,549 were women (52%) aged 52.4\pm 10.7 years and 3,211 were men (48%) aged 52.6\pm 9.9 years. Inclusion criteria Participants were aged 18–80 years. Individuals with severe or major chronic disorders, such as cancer or Parkinson’s disease, were excluded. All participants signed an informed consent form upon arrival to the research site. All identifying details of the participants were removed prior to the computational analysis. The HPP cohort study is conducted according to the principles of the Declaration of Helsinki and was approved by the Institutional Review Board (IRB) of the Weizmann Institute of Science. Detailed characteristics of the original cohort, including inclusion and exclusion criteria, have been described previously [ 39 ] . Recording protocol Speech was captured in a controlled laboratory environment using 32-bit 384 kHz sampling, then down-sampled to 16 kHz mono. Each participant performed a single 30-second counting task at a comfortable pace. No additional voice tasks were recorded. Medical labels Medical and phenotype labels were derived from the HPP questionnaire and medical records. Sleep apnea was defined

Read the complete text