MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations Soujanya Poria† , Devamanyu HazarikaΦ , Navonil Majumder‡ , Gautam Naik¶ , Erik Cambria¶ , Rada Mihalceaι † Information Systems Technology and Design, SUTD, Singapore Φ School of Computing, National University of Singapore, Singapore ‡ Centro de Investigación en Computación, Instituto Politécnico Nacional, Mexico ¶ Computer Science & Engineering, Nanyang Technological University, Singapore ι Computer Science & Engineering, University of Michigan, USA sporia@sutd.edu.sg, hazarika@comp.nus.edu.sg, navo@nlp.cic.ipn.mx, gautam@sentic.net, cambria@ntu.edu.sg, mihalcea@umich.edu Abstract Emotion recognition in conversations (ERC) is a challenging task that has recently gained popularity due to its potential applications. Until now, however, there has been no largescale multimodal multi-party emotional conversational database containing more than two speakers per dialogue. To address this gap, we propose the Multimodal EmotionLines Dataset (MELD), an extension and enhancement of EmotionLines. MELD contains about 13,000 utterances from 1,433 dialogues from the TV-series Friends. Each utterance is annotated with emotion and sentiment labels, and encompasses audio, visual, and textual modalities. We propose several strong multimodal baselines and show the importance of contextual and multimodal information for emotion recognition in conversations. The full dataset is available for use at http:// affective-meld.github.io. 1 Introduction With the rapid growth of Artificial Intelligence (AI), multimodal emotion recognition has become a major research topic, primarily due to its potential applications in many challenging tasks, such as dialogue generation, user behavior understanding, multimodal interaction, and others. A conversational emotion recognition system can be used to generate appropriate responses by analyzing user emotions (Zhou et al., 2017; Rashkin et al., 2018). Although significant research work has been carried out on multimodal emotion recognition using audio, visual, and text modalities (Zadeh et al., 2016a; Wollmer et al., 2013), significantly less work has been devoted to emotion recognition in conversations (ERC). One main reason for this is the lack of a large multimodal conversational dataset. According to Poria et al. (2019), ERC presents several challenges such as conversational context modeling, emotion shift of the interlocutors, and others, which make the task more difficult to address. Recent work proposes solutions based on multimodal memory networks (Hazarika et al., 2018). However, they are mostly limited to dyadic conversations, and thus not scalable to ERC with multiple interlocutors. This calls for a multi-party conversational data resource that can encourage research in this direction. In a conversation, the participants’ utterances generally depend on their conversational context. This is also true for their associated emotions. In other words, the context acts as a set of parameters that may influence a person to speak an utterance while expressing a certain emotion. Modeling this context can be done in different ways, e.g., by using recurrent neural networks (RNNs) and memory networks (Hazarika et al., 2018; Poria et al., 2017; Serban et al., 2017). Figure 1 shows an example where the speakers change their emotions (emotion shifts) as the dialogue develops. The emotional dynamics here depend on both the previous utterances and their associated emotions. For example, the emotion shift in utterance eight (in the figure) is hard to determine unless cues are taken from the facial expressions and the conversational history of both speakers. Modeling such complex inter-speaker dependencies is one of the major challenges in conversational modeling. Conversation in its natural form is multimodal. In dialogues, we rely on others’ facial expressions, vocal tonality, language, and gestures to anticipate their stance. For emotion recognition, multimodal- 527 Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527–536 Florence, Italy, July 28 - August 2, 2019. c 2019 Association for Computational Linguistics Neutral (Neutral) Neutral (Neutral) Anger (Negative) 1) You liked it? You really liked it? 3) Which part exactly? 5) What about the scene with the kangaroo? 7) You fell asleep! Chandler Dialogue Joey Surprise (Positive) Emotion (Sentiment) : 2) Oh, yeah! 4) The whole thing! Can we go? 6) I was surprised to see a kangaroo in a world war epic. 8) Don’t go, I’m sorry. Joy (Positive) Neutral (Neutral) Surprise (Negative) Sadness (Negative) Figure 1: Emotion shift of speakers in a dialogue in comparison with their previous emotions. Figure 2: Importance of multimodal cues. Green shows primary modalities responsible for sentiment and emotion. ity is particularly important. For the utterances with language that is difficult to understand, we often resort to other modalities, such as prosodic and visual cues, to identify their emotions. Figure 2 presents examples from the dataset where the presence of multimodal signals in addition to the text itself is necessary in order to make correct predictions of their emotions and sentiments. Multimodal emotion recognition of sequential turns encounters several other challenges. One such example is the classification of short utterances. Utterances like “yeah”, “okay”, “no” can express varied emotions depending on the context and discourse of the dialogue. However, due to the difficulty of perceiving emotions from text alone, most models resort to assigning the majority class (e.g., non-neutral in EmotionLines). Approximately 42% of the utterances in MELD are shorter than five words. We thus provide access to the multimodal data sources for each dialogue and posit that this additional information would benefit the emotion recognition task by improving the context representation and supplementing the missing or misleading signals from other modalities. Surplus information from attributes such as the speaker’s facial expressions or intonation in speech could guide models for better classification. We also provide evidence for these claims through our experiments. The development of conversational AI thus depends on the use of both contextual and multimodal information. The publicly available datasets for multimodal emotion recognition in conversations – IEMOCAP and SEMAINE – have facilitated a significant number of research projects, but also have limitations due to their relatively small number of total utterances and the lack of multi-party conversations. There are also other multimodal emotion and sentiment analysis datasets, such as MOSEI (Zadeh et al., 2018), MOSI (Zadeh et al., 2016b), and MOUD (Pérez-Rosas et al., 2013), but they contain individual narratives instead of dialogues. On the other hand, EmotionLines (Chen et al., 2018) is a dataset that contains dialogues from the popular TV-series Friends with more than two speakers. However, EmotionLines can only be used for textual analysis as it does not provide data from other modalities. In this work, we extend, improve, and further develop the EmotionLines dataset for the multimodal scenario. We propose the Multimodal EmotionLines Dataset (MELD), which includes not only textual dialogues, but also their corresponding visual and audio counterparts. This paper makes several contributions: 528 • MELD contains multi-party conversations that are more challenging to classify than dyadic variants available in previous datasets. • There are more than 13,000 utterances in MELD, which makes our dataset nearly double the size of existing multimodal conversational datasets. Dataset EmotionLines MELD • MELD provides multimodal sources and can be used in a multimodal affective dialogue system for enhanced grounded learning. dataset and MELD. all utterances from every dialogue in the EmotionLines dataset. To accomplish this, we crawl through the subtitles of all the episodes and heuristically extract the respective timestamps. In particular, we enforce the following constraints: The remainder of the paper is organized as follows: Section 2 illustrates the EmotionLines dataset; we then present MELD in Section 3; strong baselines and experiments are elaborated in Section 4; future directions and applications of MELD are covered in Section 5 and 6, respectively; finally, Section 7 concludes the paper. EmotionLines Dataset The MELD dataset has evolved from the EmotionLines dataset developed by Chen et al. (2018). EmotionLines contains dialogues from the popular sitcom Friends, where each dialogue contains utterances from multiple speakers. EmotionLines was created by crawling the dialogues from each episode and then grouping them based on the number of utterances in a dialogue into four groups of [5, 9], [10, 14], [15, 19], and [20, 24] utterances respectively. Finally, 250 dialogues were sampled randomly from each of these groups, resulting in the final dataset of 1,000 dialogues. 2.1 Annotation The utterances in each dialogue were annotated with the most appropriate emotion category. For this purpose, Ekman’s six universal emotions (Joy, Sadness, Fear, Anger, Surprise, and Disgust) were considered as annotation labels. This annotation list was extended with two additional emotion labels: Neutral and Non-Neutral. Each utterance was annotated by five workers from the Amazon Mechanical Turk (AMT) platform. A majority voting scheme was applied to select a final emotion label for each utterance. The overall Fleiss’ kappa score of this annotation process was 0.34. 3 # Utterances train dev test 10561 1178 2764 9989 1109 2610 Table 1: Comparison between the original EmotionLines • We establish a strong baseline, proposed by Majumder et al. (2019), which is capable of emotion recognition in multi-party dialogues by interparty dependency modeling. 2 # Dialogues train dev test 720 80 200 1039 114 280 Multimodal EmotionLines Dataset (MELD) 1. Timestamps of the utterances in a dialogue must be in an increasing order. 2. All the utterances in a dialogue have to belong to the same episode and scene. These constraints revealed a few outliers in EmotionLines where some dialogues span across scenes or episodes. For example, the dialogue in Table 2 contains two natural dialogues from episode 4 and 20 of season 6 and 5, respectively. We decided to filter out these anomalies, thus resulting in a different number of total dialogues in MELD as compared to EmotionLines (see Table 1). Next, we employ three annotators to label each utterance, followed by a majority voting to decide the final label of the utterances. We drop a few utterances where all three annotations were different, and also remove their corresponding dialogues to maintain coherence. A total of 89 utterances spanning 11 dialogues fell under this category. Finally, after obtaining the timestamp of each utterance, we extract their corresponding audiovisual clips from the source episode followed by the extraction of audio content from these clips. We format the audio files as 16-bit PCM WAV files for further processing. The final dataset includes visual, audio, and textual modalities for each utterance.1 3.1 Dataset Re-annotation The utterances in the original EmotionLines dataset were annotated by looking only at the transcripts. However, due to our focus on multimodality, we re-annotate all the utterances by asking the three annotators to also look at the available video clip of the utterances. We then use majority-voting to obtain the final label for each utterance. 1 We start the construction of the MELD corpus by extracting the starting and ending timestamps of 529 We consulted a legal office to verify that the usage and distribution of very short length videos fall under the fair use category. S5.E20 S6.E4 Episode Utterance What are you talkin about? I never left you! Youve always been my agent! Really?! Yeah! Oh well, no harm, no foul. Okay, you guys free tonight? Yeah!! Tonight? You-you didn’t say it was going to be at nighttime. Speaker Joey Estelle Joey Estelle Gary Ross Chandler Emotion surprise surprise joy neutral neutral joy surprise Sentiment negative positive positive neutral neutral positive negative Table 2: A dialogue in EmotionLines where utterances from two different episodes are present. The first four utterances in this dialogue have been taken from episode 4 of season 6. The last three utterances in red font are from episode 20 of season 5. Utterance You know? Forget it! Oh no-no, give me some specifics. I was surprised to see a kangaroo in a World War epic. Or, call an ambulance. Annotator 1 sadness Annotator 2 disgust Annotator 3 anger neutral anger sadness surprise anger joy anger surprise neutral Emotion Categories anger disgust fear joy neutral sadness surprise negative neutral positive Sentiment The annotators were graduate students with high proficiency in English speaking and writing. Before starting the annotation, they were briefed about the annotation process with a few examples. We achieve an overall Fleiss’ kappa score of 0.43 which is higher than the original EmotionLines annotation whose kappa score was 0.34 (kappa of IEMOCAP annotation process was 0.4), thus suggesting the usefulness of the additional modalities during the annotation process. 2,772 utterances in the EmotionLines dataset were labeled as non-neutral where the annotators agreed that the emotion is not neutral but they could not reach agreement regarding the correct emotion label. This hampers classification, as the non-neutral utterance space and the other emotionlabel spaces get conflated. In our case, we remove the utterances where the annotators fail to reach an agreement on the definite emotion label. The number of disagreements in our annotation process is 89, which is much lower than the 2,772 disagreements in EmotionLines, reflecting again the annotation improvement obtained through a multimodal dataset. Table 3 shows examples of utterances where the annotators failed to reach consensus. Table 4 shows the label-wise comparison between EmotionLines and MELD dataset. For most of the utterances in MELD, the annotations match the original annotations in EmotionLines. Yet, there exists a significant amount of samples whose utterances have been changed in the re-annotation process. For example, the utterance This guy fell asleep! (see Table 5), was labeled as non-neutral EmotionLines Train Dev Test 524 85 163 244 26 68 190 29 36 1283 123 304 4752 491 1287 351 62 85 1221 151 286 - Train 1109 271 268 1743 4710 683 1205 2945 4710 2334 MELD Dev Test 153 345 22 68 40 50 163 402 470 1256 111 208 150 281 406 833 470 1256 233 521 Table 4: Emotion and Sentiment distribution in MELD vs. EmotionLines. in EmotionLines but after viewing the associated video clip, it is correctly re-labeled as anger in MELD. The video of this utterance reveals an angry and frustrated facial expression along with a high vocal pitch, thus helping to recognize its correct emotion. The annotators of EmotionLines had access to the context, but this was not sufficient, as the availability of additional modalities can sometime bring more information for the classification of such instances. These scenarios justify both context and multimodality to be important aspects for emotion recognition in conversation. Timestamp alignment. There are many utterances in the subtitles that are grouped within identical timestamps in the subtitle files. In order to find the accurate timestamp for each utterance, we use a transcription alignment tool Gentle,2 which automatically aligns a transcript with the audio by extracting word-level timestamps from the audio (see Table 6). In Table 7, we show the final format of the MELD dataset. Table 3: Some examples of the utterances for which annotators could not reach consensus. Dyadic MELD. We also provide another version of MELD where all the non-extendable contiguous dyadic sub-dialogues of MELD are extracted. For example, let a three-party dialogue in MELD with speaker ids 1, 2, 3 have their turns in the following 2 530 http://github.com/lowerquality/gentle order: [1, 2, 1, 2, 3, 2, 1, 2]. From this dialogue sequence, dyadic MELD will have the following sub-dialogues as samples: [1, 2, 1, 2], [2, 3, 2] and [2, 1, 2]. However, the reported results in this paper are obtained using only the multiparty variant of MELD. Utterance I’m so sorry! Look! This guy fell asleep! Speaker Chandler Chandler Chandler MELD sadness surprise anger EmotionLines sadness surprise non-neutral Table 5: Difference in annotation between EmotionLines and MELD. 3.2 Dataset Exploration As mentioned before, we use seven emotions for the annotation, i.e., anger, disgust, fear, joy, neutral, sadness, and surprise, across the training, development, and testing splits (see Table 4). It can be seen that the emotion distribution in the dataset is expectedly non-uniform with the majority emotion being neutral. We have also converted these fine-grained emotion labels into more coarse-grained sentiment classes by considering anger, disgust, fear, sadness as negative, joy as positive, and neutral as neutral sentiment-bearing class. Surprise is an example of a complex emotion which can be expressed with both positive and negative sentiment. The three annotators who performed the utterance annotation further annotated the surprise utterances into either positive or negative sentiment classes. The entire sentiment annotation task reaches a Fleiss’ kappa score of 0.91. The distribution of positive, negative, neutral sentiment classes is given in Table 4. Table 8 presents several key statistics of the dataset. The average utterance length – i.e. number of words in an utterance – is nearly the same across training, development, and testing splits. On average, three emotions are present in each dialogue of the dataset. The average duration of an utterance is 3.59 seconds. The emotion shift of a speaker in a dialogue makes emotion recognition task very challenging. We observe that the number of such emotion shifts in successive utterances of a speaker in a dialogue is very frequent: 4003, 427, and 1003 in train/dev/test splits, respectively. Figure 1 shows an example where speaker’s emotion changes with time in the dialogue. across the emotion and sentiment labels, respectively. Figure c shows the overall coverage of the speakers across the dataset. Multiple infrequent speakers (< 1% utterances) are grouped as Others. 3.3 Related Datasets Most of the available datasets in multimodal sentiment analysis and emotion recognition are nonconversational. MOSI (Zadeh et al., 2016b), MOSEI (Zadeh et al., 2018), and MOUD (Pérez-Rosas et al., 2013) are such examples that have drawn significant interest from the research community. On the other hand, IEMOCAP and SEMAINE are two popular dyadic conversational datasets where each utterance in a dialogue is labeled by emotion. The SEMAINE Database is an audiovisual database created for building agents that can engage a person in a sustained and emotional conversation (McKeown et al., 2012). It consists of interactions involving a human and an operator (either a machine or a person simulating a machine). The dataset contains 150 participants, 959 conversations, each lasting around 5 minutes. A subset of this dataset was used in AVEC 2012’s fully continuous sub-challenge (Schuller et al., 2012) that requires predictions of four continuous affective dimensions: arousal, expectancy, power, and valence. The gold annotations are available for every 0.2 second in each video for a total of 95 videos comprising 5, 816 utterances. The Interactive Emotional Dyadic Motion Capture Database (IEMOCAP) consists of videos of dyadic conversations among pairs of 10 speakers spanning 10 hours of various dialogue scenarios (Busso et al., 2008). Videos are segmented into utterances with annotations of fine-grained emotion categories: anger, happiness, sadness, neutral, excitement, and frustration. IEMOCAP also provides continuous attributes: activation, valence, and dominance. These two types of discrete and continuous emotional descriptors facilitate the complementary insights about the emotional expressions of humans and emotional communications between people. The labels in IEMOCAP were annotated by at least three annotators per utterance and self-assessment manikins (SAMs) were also employed to evaluate the corpus (Bradley and Lang, 1994). 3.4 Character Distribution. In Figure 3, we present the distributional details of the primary characters in MELD. Figure a and b illustrate the distribution Comparison with MELD Both resources mentioned above are extensively used in this field of research and contain settings 531 Utterance Chris says they’re closing down the bar. No way! Season Incorrect Splits Start Time End Time Episode Corrected Splits Start Time End Time 3 6 00:05:57,023 00:05:59,691 00:05:57,023 00:05:58,734 3 6 00:05:57,023 00:05:59,691 00:05:58,734 00:05:59,691 Table 6: Example of timestamp alignment using the Gentle alignment tool. Utterance But then who? The waitress I went out with last month? You know? Forget it! Speaker Emotion D ID U ID Season Episode StartTime Joey Rachel EndTime surprise 1 0 9 23 00:36:40,364 00:36:42,824 sadness 1 1 9 23 00:36:44,368 00:36:46,578 Table 7: MELD dataset format for a dialogue. Notations: D ID = dialogue ID, U ID = utterance ID. StartTime and EndTime are in hh:mm:ss,ms format. that are aligned to the components of MELD. However, MELD is different in terms of both complexity and quantity. Both IEMOCAP and SEMAINE contain dyadic conversations, wherein the dialogues in MELD are multi-party. Multi-party conversations are more challenging compared to dyadic. They provide a flexible setting where multiple speakers can engage. From a research perspective, such availability also demands proposed dialogue models to be scalable towards multiple speakers. MELD also includes more than 13000 emotion labeled utterances, which is nearly double the annotated utterances in IEMOCAP and SEMAINE. Table 9 provides information on the number of available dialogues and their constituent utterances for all three datasets, i.e., IEMOCAP, SEMAINE, and MELD. Table 10 shows the distribution for common emotions as well as highlights a few key statistics of IEMOCAP and MELD. 4 Experiments 4.1 Feature Extraction We follow Poria et al. (2017) to extract features for each utterance in MELD. For textual features, we initialize each token with pre-trained 300-dimensional GloVe vectors (Pennington et al., 2014) and feed them to a 1D-CNN to extract 100 MELD Statistics # of modalities # of unique words Avg./Max utterance length # of dialogues # of dialogues dyadic MELD # of utterances # of speakers Avg. # of utterances per dialogue Avg. # of emotions per dialogue Avg./Max # of speakers per dialogue # of emotion shift Avg. duration of an utterance Train {a,v,t} 10,643 8.0/69 1039 2560 9989 260 9.6 3.3 2.7/9 4003 3.59s Dev {a,v,t} 2,384 7.9/37 114 270 1109 47 9.7 3.3 3.0/8 427 3.59s Test {a,v,t} 4,361 8.2/45 280 577 2610 100 9.3 3.2 2.6/8 1003 3.58s dimensional textual features. For audio, we use the popular toolkit openSMILE (Eyben et al., 2010), which extracts 6373 dimensional features constituting several low-level descriptors and various statistical functionals of varied vocal and prosodic features. As the audio representation is high dimensional, we employ L2-based feature selection with sparse estimators, such as SVMs, to get a dense representation of the overall audio segment. For the baselines, we do not use visual features, as videobased speaker identification and localization is an open problem. Bimodal features are obtained by concatenating audio and textual features. 4.2 Baseline Models To provide strong benchmarks for MELD, we perform experiments with multiple baselines. Hyperparameter details for each baseline can be found at http://github.com/senticnet/meld. text-CNN applies CNN to the input utterances without considering the context of the conversation (Kim, 2014). This model represents the simplest baseline which does not leverage context or multimodality in its approach. bcLSTM is a strong baseline proposed by Poria et al. (2017), which represents context using a bi-directional RNN. It follows a two-step hierarchical process that models uni-modal context first and then bi-modal context features. For unimodal text, a CNN-LSTM model extracts contextual representations for each utterance taking the GloVe em- Dataset IEMOCAP SEMAINE MELD Type acted acted acted # dialogues train dev test 120 31 58 22 1039 114 280 # utterances train dev test 5810 1623 4386 1430 9989 1109 2610 Table 9: Comparison among IEMOCAP, SEMAINE, and Table 8: Dataset Statistics. {a,v,t} = {audio, visual, text} proposed MELD datasets 532 Neutral Sadness Disgust Joy Anger Positive 750 0 0 C ha O Ra Jo ic oe on M Ph ha Neutral Others Chandler 11% 16% Ross 14% Rachel 16% Phoebe 13% Joey Monica 16% 14% ch el th er s 750 a 1500 ey 1500 be 2250 nd le r Ro ss 2250 C Negative 3000 nd le Ro r ss Ph oe b M e on ic a Jo ey Ra ch e O l th er s 3000 Surprise Fear a) Character-Emotion b) Character-Sentiment c) Character-distribution Figure 3: Character distribution across MELD. Emotions Dataset IEMOCAP MELD Happy/Joy Anger Disgust Sadness Surprise Neutral 648 2308 1103 1607 2 361 1084 1002 107 1636 1708 6436 Avg. utterence length 15.8 8.0 Other Statistics #Unique words 3,598 10,643 Avg. conversation length 49.2 9.6 Table 10: Comparison among IEMOCAP and proposed MELD datasets. beddings as input. For unimodal audio, an LSTM model gets audio representations for each audio utterance feature vector. Finally, the contextual representations from the unimodal variants are supplied to the bimodal model for classification. bcLSTM does not distinguish among different speakers and models a conversation as a single sequence. DialogueRNN represents the current state of the art for conversational emotion detection (Majumder et al., 2019). It is a strong baseline with effective mechanisms to model context by tracking individual speaker states throughout the conversation for emotion classification. DialogueRNN is capable of handling multi-party conversation so it can be directly applied on MELD. It employs three stages of gated recurrent units (GRU) (Chung et al., 2014) to model emotional context in conversations. The spoken utterances are fed into two GRUs: global and party GRU to update the context and speaker state, respectively. In each turn, the party GRU updates its state based on 1) the utterance spoken, 2) the speaker’s previous state, and 3) the conversational context summarized by the global GRU through an attention mechanism. Finally, the updated speaker state is fed into the emotion GRU which models the emotional information for classification. Attention mechanism is used on top of the emotion GRU to leverage contextual utterances by different speakers at various distances. To analyze the role of multimodal signals, we analyze DialogueRNN and bcLSTM on MELD for both uni and multimodal settings. Training involved usage of class weights to alleviate imbalance issues. 4.3 Results We provide results for the two tasks of sentiment and emotion classification on MELD. Table 13 shows the performance of sentiment classification by using DialogueRNN, whose multimodal variant achieves the best performance (67.56% F-score) surpassing multimodal bcLSTM (66.68% F-score). Multimodal DialogueRNN also outperforms its unimodal counterparts. However, the improvement due to fusion is about 1.4% higher than the textual modality which suggests the possibility of further improvement through better fusion mechanisms. The textual modality outperforms the audio modality by about 17%, which indicates the importance of spoken language in sentiment analysis. For positive sentiment, audio modality performs poorly. It would be interesting to analyze the clues specific to positive sentiment bearing utterances in MELD that the audio modality could not capture. Future work should aim for enhanced audio feature extraction schemes to improve the classification performance. Table 11 presents the results of the baseline models on MELD emotion classification. The performance on the emotion classes disgust, fear, and sadness are particularly poor. The primary reason for this is the inherent imbalance in the dataset which has fewer training instances for these mentioned emotion classes (see Table 4). We partially tackle this by using class-weights as hyper-parameters. Yet, the imbalance calls for further improvement for future work to address. We also observe high 533 Models text-CNN cMKL text+audio text bcLSTM audio text+audio text DialogueRNN audio text+audio anger 34.49 39.50 42.06 25.85 43.39 40.59 35.18 43.65 disgust 8.22 16.10 21.69 6.06 23.66 2.04 5.13 7.89 fear 3.74 3.75 7.75 2.90 9.38 8.93 5.56 11.68 Emotions joy neutral 49.39 74.88 51.39 72.73 54.31 71.63 15.74 61.86 54.48 76.67 50.27 75.75 13.17 65.57 54.40 77.44 sadness 21.05 23.95 26.92 14.71 24.34 24.19 14.01 34.59 surprise 45.45 46.25 48.15 19.34 51.04 49.38 20.47 52.51 w-avg. 55.02 55.51 56.44 39.08 59.25 57.03 41.79 60.25 Table 11: Test-set weighted F-score results of DialogueRNN for emotion classification in MELD. Note: w-avg denotes weighted-average. text-CNN and cMKL: contextual information were not used. mis-classification rate between the anger, disgust, and fear emotion categories as these emotions have subtle differences among them causing harder disambiguation. Similar to sentiment classification trends, the textual classifier outperforms (57.03% F-score) the audio classifier (41.79% F-score). Multimodal fusion helps in improving the emotion recognition performance by 3%. However, multimodal classifier performs worse than the textual classifier in classifying sadness. To analyze further, we also run experiments on 5-class emotions by dropping the infrequent fear and disgust emotions (see Table 12). Not surprisingly, the results improve over the 7-class setting with significantly better performance by the multimodal variant. Overall, emotion classification performs poorer than sentiment classification. This observation is expected as emotion classification deals with classification with more fine-grained classes. 4.4 tion dynamics is the influence of fellow speakers in the multi-party setting. We analyze this factor by looking at the activation of the attention module on the global GRU in DialogueRNN. We observe that in 63% (882/1381) of the correct test predictions, the highest historical attention is given to utterances from different speakers. This significant proportion suggests inter-speaker influence to be an important parameter. Unlike DialogueRNN, Mode bcLSTM dRNN∗ Emotions ang joy neu sad bcLSTM T+A 45.9 52.2 77.9 11.2 T 41.7 53.7 77.8 21.2 A 34.1 18.8 66.2 16.0 dRNN∗ T+A 48.2 53.2 77.7 20.3 ∗ dRNN: DialogueRNN, T: text, A: audio surp 49.9 47.7 16.6 48.5 w-avg. 60.6 60.8 44.3 61.6 Table 12: Test-set weighted F-score results of DialogueRNN for 5-class emotion classification in MELD. Note: w-avg denotes weighted-average. surp: surprise emotion. w-avg. 64.25 66.68 66.10 49.61 67.56 for sentiment classification in MELD. bcLSTM does not utilize speaker information while detecting emotion. Table 11 shows that in all the experiments, DialogueRNN outperforms bcLSTM by 1-2% margin. This result supports the claim by Majumder et al. (2019) that speaker-specific modeling of emotion recognition is beneficial as it helps in improving context representation and incorporates important clues such as inter-speaker relations. Role of Context. One of the main purposes of MELD is to train contextual modeling in a conversation for emotion recognition. Table 11 and 13 show that the improvement over the non-contextual model such as text-CNN – which only uses a CNN (see Section 4.1) – is 1.4% to 2.5%. Mode Sentiments neg. neu. 55.42 74.69 57.87 60.04 60.10 74.94 45.53 62.33 58.18 78.40 Table 13: Test set weighted F-score results of DialogueRNN Additional Analysis Inter-speaker influence. One of the important considerations while modeling conversational emo- text-CNN T+A T A T+A pos. 53.23 74.68 54.35 25.47 54.29 Emotion shifts. The ability to anticipate the emotion shifts within speakers throughout the course of a dialogue has synergy with better emotion classification. In our results, DialogueRNN achieves a recall of 66% for detecting emotion shifts. However, in the ideal scenario, we would want to detect shift along with the correct emotion class. For this setting, DialogueRNN gets a recall of 36.7%. The deterioration observed is expected as solving both tasks together has a higher complexity. Future methods would need to improve upon their capabilities of detecting shifts to improve the emotion 534 classification. Contextual distance. Figure 4 presents the distribution of distances between the target utterance and its second highest attended utterance within the conversation by DialogueRNN in its emotion GRU. For the highest attention, the model largely focuses on utterances nearby to the target utterance. However, the dependency on distant utterances increases with the second highest attention. Moreover, it is interesting to see that the dependency exists both towards the historical and the future utterances, thus incentivizing utilization of bi-directional models. 5 Future Directions Frequency of correct predictions Future research using this dataset should focus on improving contextual modeling. Helping models reason about their decisions, exploring emotional influences, and identifying emotion shifts are promising aspects. Another direction is to use visual information available in the raw videos. Identifying face of the speaker in a video where multiple other persons are present is very challenging. This is the case for MELD too as it is a multi-party 1200 Distance between test utterance and 1000 2nd highest attention We believe there is room for further improvement using other more advanced fusion methods such as MARN (Zadeh et al., 2018). 6 MELD has multiple use-cases. It can be used to train emotion classifiers to be further used as emotional receptors in generative dialogue systems. These systems can be used to generate empathetic responses (Zhou et al., 2017). It can also be used for emotion and personality modeling of users in conversations (Li et al., 2016). By being multimodal, MELD can also be used to train multimodal dialogue systems. Although by itself it is not large enough to train an end-to-end dialogue system (Table 1), the procedures used to create MELD can be adopted to generate a largescale corpus from any multimodal source such as popular sitcoms. We define multimodal dialogue system as a platform where the system has access to the speaker’s voice and facial expressions which it exploits to generate responses. Multimodal dialogue systems can be very useful for real time personal assistants such as Siri, Google Assistant where the users can use both voice and text and facial expressions to communicate. 7 Highest attention 800 600 400 200 0 15 10 5 0 5 10 15 Δt Figure 4: Histogram of ∆t = distance between the target and its context utterance based on emotion GRU attention scores. dataset. Enhancements can be made by extracting relevant visual features through processes utilizing audio-visual speaker diarization. Such procedures would enable utilizing a visual modality in the baselines. In our results, audio features do not help significantly. Thus, we believe that it is necessary to improve the feature extraction for these auxiliary modalities in order to improve the performance further. So far, we have only used concatenation as a feature fusion approach, and showed that it outperforms the unimodal baselines by about 1-3%. Applications of MELD Conclusion In this work, we introduced MELD, a multimodal multi-party conversational emotion recognition dataset. We described the process of building this dataset, and provided results obtained with strong baseline methods applied on this dataset. MELD contains raw videos, audio segments, and transcripts for multimodal processing. Additionally, we also provide the features used in our baseline experiments. We believe this dataset will also be useful as a training corpus for both conversational emotion recognition and multimodal empathetic response generation. Building upon this dataset, future research can explore the design of efficient multimodal fusion algorithms, novel ERC frameworks, as well as the extraction of new features from the audio, visual, and textual modalities.