2M-Belebele : Highly Multilingual Speech and American Sign Language Comprehension Dataset

Marta R. Costa-jussà, Bokai Yu, Pierre Andrews, Belen Alastruey, Necati Cihan Camgoz · 2024

Read 2M-Belebele : Highly Multilingual Speech and American Sign Language Comprehension Dataset by Marta R. Costa-jussà, Bokai Yu, Pierre Andrews, Belen Alastruey, Necati Cihan Camgoz (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08274.txt.

Introduction

Marta Costa-jussà and co-authors release a comprehension data set spanning speech in many languages plus American Sign Language. It grows a reading benchmark by adding matching speech, covering seventy-four spoken languages and one sign language. Models were tested in few-shot and zero-shot modes. Speech scores ran about two to three percent below reading scores on average, the abstract reports.

Catalog record

CREATORMarta R. Costa-jussà, Bokai Yu, Pierre Andrews, Belen Alastruey, Necati Cihan Camgoz
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08274

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

2M-Belebele : Highly Multilingual Speech and American Sign Language Comprehension Dataset Marta R. Costa-jussà, Bokai Yu, Pierre Andrews, Belen Alastruey, Necati Cihan Camgoz Abstract We introduce the first highly multilingual speech and American Sign Language (ASL) comprehension dataset by extending Belebele . Our dataset covers 74 spoken languages at the intersection of Belebele and FLEURS , and one sign language (ASL). We evaluate 2M-Belebele dataset for both 5-shot and zero-shot settings and across languages, the speech comprehension accuracy is ≈ \approx 2-3% average lower compared to reading comprehension. 1 Introduction From an AI perspective, text understanding and generation services are used globally in more than a hundred languages, but the scarcity of labeled data poses a significant challenge to developing functional systems in most languages. Although natural language processing (NLP) datasets with extensive language coverage, such as FLORES-200 ( NLLB-Team et al. 2022 ) , are available, they mainly concentrate on machine translation (MT). Multilingual evaluation benchmarks such as those for multilingual question answering ( Lewis et al. 2020 ; Clark et al. 2020 ) , natural language inference ( Conneau et al. 2018 ) , summarization ( Hasan et al. 2021 ; Ladhak et al. 2020 ) , and reasoning datasets ( Ponti et al. 2020 ; Lin et al. 2021 ) collectively cover only about 30 languages. Furthermore, the extension of such datasets to speech or American Sign Language (ASL) is lacking, with the exception of FLEURS ( Conneau et al. 2022 ; Tanzer 2024 ) , which is based on FLORES-200 . The recent Belebele benchmark is the first corpus that addresses text reading comprehension for a large number of languages following a multi-way parallel approach ( Bandarkar et al. 2023 ) . The entire Belebele text statistics are summarized in Table 1 . Currently, there are no highly multilingual evaluation datasets for natural language understanding that cover either both speech and text, and/or ASL. The outstanding performance of some MT and text-to-speech (TTS) models has enabled a rise in the number of works using synthetically generated training data. Furthermore, some recent works propose to also use synthetic data for evaluation; e.g., Üstün et al. 2024 ; Seamless-Communication et al. 2023 ; Nguyen et al. 2024 ; Nachmani et al. 2023 . This strategy allows researchers to extend datasets to low-resource languages and to other modalities, such as speech. However, we prove that using synthetic data for evaluation does not provide comparable conclusions as relying on human speech for the particular task of automatic speech recognition (ASR) and the FLEURS domain (see Appendix D ). The evaluation dataset that is closest to the speech comprehension evaluation dataset presented in this paper is the generative QA dataset proposed in Nachmani et al. 2023 . The questions are taken from two sources: the WebQuestions dataset created by Berant et al. 2013 and a new test set called “LLama Questions.” The dataset covers 300 questions in English. In this work, we extend the Belebele dataset to speech and sign (Section 2 ). By doing so, we create the first highly multilingual speech and sign comprehension dataset 1 1 1 Appendix A specifies language coverage. : 2M-Belebele . Compared to spoken languages, sign languages are considered low-resource languages for natural language processing ( Yin et al. 2021 ) . Most popular datasets cover small domains of discourse; e.g., weather broadcasts ( Camgoz et al. 2018 ) , which has limited real world applications. There have been previous releases of large scale open domain sign language datasets; e.g., Albanie et al. 2021 ; Shi et al. 2022 ; Uthus et al. 2024 . However, the results and challenges on such datasets suggest that computational sign language research still requires additional datasets to reach the performance of their spoken language counterparts ( Müller et al. 2022 ; Müller et al. 2023 ) . With the release of the ASL extension of the Belebele dataset, we aim to provide additional, high quality sign language data with gloss annotations to underpin further computational sign language research. Furthermore, due to the paragraph-level nature of the Belebele dataset, we enable paragraph-context sign language translation, which has been reported to improve translation performance ( Sincan et al. 2023 ) . 2M-Belebele is composed of human speech recordings covering 74 languages and human sign recordings for ASL. As a by-product of 2M-Belebele , we also extend the FLEURS dataset (which is widely used to benchmark language identification and ASR) by providing recordings for more FLORES-200 sentences than were previously available and adding sign language, creating a new 2M-FLORES . This 2M-FLORES extends FLEURS by 20%. Finally, we provide a very basic set of experiments that evaluate 2M-Belebele and provide some reference results on the dataset. We use direct and/or cascaded systems to evaluate 2M-Belebele dataset with direct and/or cascaded systems (Section 3 ). We also list several further experimentation that 2M-Belebele unblocks. Note that the main contribution of this paper is the creation of the first highly multilingual speech and sign comprehension dataset. The complete set of experiments is out of the scope of this paper (see Section on Limitations). By open-sourcing our dataset, we encourage the scientific community to pursue such experimentation. 2 2M-Belebele Passages Questions/Answers Distinct Passages 488 Distinct Q 900 Questions per passage 1-2 Multiple-choice A 4 Avg words (std) 79.1 (26.2) Avg words Q (std) 12.9 (4.0) Avg sentences (std) 4.1 (1.4) Avg words A (std) 4.2 (2.9) Table 1: Statistics for 2M-Belebele , which covers 74 spoken languages plus ASL. Average words are computed for English. FLEURS and Belebele passage alignment. Since Belebele uses passages constructed from sentences in the FLORES-200 dataset, and FLEURS ( Conneau et al. 2022 ) is a human speech version of FLORES-200 for a subset of its languages, we create a speech version of Belebele by aligning its passages with the speech segments available in FLEURS . This extension can be done without extra human annotation, just by computing the alignment between FLEURS and Belebele passages. However, such alignment does not cover the entire Belebele corpus because FLEURS does not cover the entirety of FLORES-200 . There are 74 languages shared between FLEURS and Belebele . FLEURS does not cover the same passages as Belebele in all those 74 languages, which means that some languages have more speech passages than others. In general, we are able to match almost \approx 80% of the passages. Figure 1 shows the number of FLEURS paragraphs we can match, thus obtaining the number of paragraphs that must be recorded in order to cover all passages Belebele . Figure 1: FLEURS vs New Recordings from 2M-Belebele for sentences in passages. Speech recordings. We commission human recordings for the part of the Belebele dataset that is not covered by existing FLEURS recordings, as well as for elements of Belebele that do not exist in FLEURS (i.e. questions and answers). Recording participants must be native speakers of the languages they record. They must have an impeccable grasp of the conventions used in their respective languages for the narration of texts. The three tasks that participants are asked to perform are: (1) Read aloud and record the text passages provided (from FLORES-200 ); (2) Read aloud and

Read the complete text