Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Research

2026

Read Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25776.txt.

Introduction

Wong and three colleagues survey speech emotion recognition research to test whether stated motivations match actual practice. Critical analyses have raised ethical concerns about task validity and downstream impacts, urging alignment between goals and methods. The survey finds appealing goals, such as voice-activated systems and healthcare uses, but datasets that do not reflect those deployment settings. That gap between motivation and practice, the authors argue, breeds ethical concerns. They call for concrete use cases to prevent misinterpretation, misuse, and downstream harm.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36309
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25776

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Research Abstract Critical analyses of emotion recognition technology have raised ethical concerns around task validity and potential downstream impacts, urging researchers to ensure alignment between their stated motivations and practice. However, these discussions have not adequately influenced or drawn from research on speech emotion recognition (SER). We address this gap by conducting a systematic survey of SER research to uncover what stated motivations drive this work and if they align with the datasets and emotions studied. We find that while SER research identifies appealing goals—such as well-situated voice-activated systems or healthcare applications—commonly-used datasets do not reflect these proposed deployment contexts, thus presenting a gap between motivations and research practices. We argue that such gaps engender ethical concerns, and that SER research should reassert itself with concrete use-cases to prevent misinterpretations, misuse, and downstream harms. Keywords: emotion recognition, critical survey, ethics Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Research Taryn Wong 1 , Zeerak Talat 2 , Hanan Aldarmaki 3 , Anjalie Field 1 1 Johns Hopkins University, 2 University of Edinburgh, 3 MBZUAI [email protected], [email protected], [email protected] Abstract content 1. Introduction Emotion AI, including the detection of human emotions from video, image, audio, or text data using machine learning (ML) methods, has become a popular research area with increasing commercialization in a broad range of applications McStay 2018 . The growing adoption of this technology has raised concerns around its development and use. Researchers have questioned the overall task validity, e.g., the premise of using someone’s outer appearance to infer inner character or state, particularly given the lack of consensus on how to define emotions among psychologists McStay and Pavliscak 2019 ; Stark and Hoey 2021 . Empirical qualitative studies have demonstrated that people view emotion AI as invasive and privacy-violating Andalibi and Buss 2020 ; Pyle et al. 2024 , and its use in contexts like workplaces or hiring processes create emotional labor with little perceived benefit for those subjected to it Roemmich et al. 2023 ; Pyle et al. 2024 . Importantly, concerns raised are often specific to emotion AI, rather than technology more generally Pyle et al. 2024 . To date, this work has had limited engagement with the abundance of research on Speech Emotion Recognition (SER). Research on task validity Jacobs and Wallach 2021 and ethics have focused on facial recognition or broadly construed AI systems where underlying data and models are unspecified Domnich and Anbarjafari 2021 ; Pyle et al. 2024 ; Roemmich et al. 2023 ; Karizat et al. 2024 . While some studies have questioned the ethics and validity of SER ( Katirai 2023 ; Hartmann et al. 2024 ; Morozov 2014 ; Testa et al. 2023 ; Juslin et al. 2005 ) , no prior work has conducted a systematic investigation of SER research practices. In this work, we follow the suggestion proposed by Stark and Hoey 2021 that “[d]esigners and developers should think twice before embarking on emotion AI projects” and we investigate to what extent SER research meets their proposed “necessary though not sufficient condition” that projects have “clear alignment between conceptual models, data, norms, and aims”. We formalize our study into three research questions: (1) What do SER researchers state as their motivations? (2) What purposes do popular datasets support? (3) Do datasets align with stated motivations? RQ1 focuses on why researchers work on SER, while RQ2 focuses on how . RQ3 assesses the alignment between the why and how . We investigate these questions by conducting a systematic survey of 88 SER research papers, which we manually coded for stated motivations, specific emotions studied, and datasets used. We ultimately find a substantial gap between stated motivations and experimental setups, and we propose ways to reduce this gap by seeking suitable datasets or pursuing different motivations. Our work encourages future researchers to “think twice” about SER projects, and to seek alignment between objectives, research practices, and potential implications, rather than viewing SER as an isolated research task. 2. Methodology Our work uses similar systemic survey methodology as previous studies that reflect on practices in speech processing research and related disciplines Blodgett et al. 2020 ; Field et al. 2021 ; Birhane et al. 2022 ; Raff et al. 2023 . First, we queried Semantic Scholar for papers containing search terms ‘‘speech’’ and ‘‘emotion’’ and {‘‘recognition’’, ‘‘classification’’, ‘‘data’’, or ‘‘database’’} in their title or abstract, collecting a total of 7,486 after filtering out papers without publication venue information. Of the initial sample, we kept only those published in popular archival speech, natural language processing, or ML venues, which we determined by manually reviewing venues with the most papers in the initial sample, 1 1 1 ICASSP; ASRU; SLT; Interspeech; NeurIPS; ICLR; ICML; AAAI; IEEE/ACM Transactions on Audio, Speech, and Language Processing; IEEE Open Journal of Signal Processing; IEEE Journal of Selected Topics in Signal Processing; Computer Speech & Language; Speech Communication; IEEE Transactions on Affective Computing; JMLR; ACL resulting in 959 papers. We then randomly sampled 108 (stratified by publication year and venue) for in-depth manual analysis. We used a standard inductive coding procedure to label the sampled 108 papers. First, three authors independently coded a set of 10 papers each (30 papers total), where each author first determined if the paper consisted of SER. Then, each author labeled the paper’s stated motivations, specific emotions investigated, and data sets used in experiments or analysis. 2 2 2 We also initially coded acknowledged funding sources, but found they were too sparse to analyze in aggregate. In labeling stated motivations, annotators also quoted the exact text from the paper describing these motivations. The annotators then discussed the labeling scheme and finalized a set of frequently-stated motivations for which to code. Annotators then re-coded the initial 30 documents under the revised scheme, and then each independently coded an additional set of 10 documents to ensure mutual understanding of the coding scheme. The remainder of the data was then singly coded. Finally, the annotators re-convened and discussed further revisions to the scheme, specifically adding categories of motivations that were found in the larger data sample and re-coded the data under the revised scheme. Throughout this process, we removed 20 papers whose primary focus was not SER (e.g., papers that focused on depression detection, speech generation, speech representation, etc.), leaving a final set of 88 papers. The final set primarily consists of papers published in speech venues (Interspeech: 50 papers, ICASSP: 18, ASRU: 1, SLT: 2, Speech Communication: 4), as well as 9 papers from IEEE Transactions on Affective Computing. The final coding for all 88 papers is in Table 5 in the Appendix. 3. Results What do SER researchers state as their motivations? Responsive bots: other HCI systems 42.05% Responsive bots: voice assistants 12.50% Responsive bots: car voice assistants 6.82% Healthcare (mental health) 18.18% Call screening 17.05% Video games, toys, entertainment 13.64% Education 9.09% Paralinguistics / behavioral studies 6.82% Social companion bots 4.55% Lie detection 3.41% Prior work 27.27% Other 14.77% Table 1: Percent of manually coded papers that reference each motivation. Motivations are grouped thematically. Percents do not add to 100 as papers often reference

Read the complete text