An Overview and Discussion of the Suitability of Existing Speech Datasets to Train Machine Learning Models for Collective Problem Solving

Gnaneswar Villuri, Alex Doboli · 2024

Read An Overview and Discussion of the Suitability of Existing Speech Datasets to Train Machine Learning Models for Collective Problem Solving by Gnaneswar Villuri, Alex Doboli (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18489.txt.

Introduction

Gnaneswar Villuri and Alex Doboli judge whether speech datasets built for other tasks can train team problem-solving models. Their 2024 report defines the target first: small teams of three or four talking through a shared problem. A dataset here means recordings of such talk.

Metrics were drawn up for the mental, social, and emotional strands of teamwork. A large family of spoken-language datasets was then scored against them. Most were built for single speakers or commands, far from joint problem talk. The report lists what a purpose-built dataset would need. The gap it maps should guide future collection work.

Catalog record

CREATORGnaneswar Villuri, Alex Doboli
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE96 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18489

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Overview and Discussion of the Suitability of Existing Speech Datasets to Train Machine Learning Models for Collective Problem Solving Gnaneswar Villuri, Alex Doboli Abstract This report characterized the suitability of existing datasets for devising new Machine Learning models, decision making methods, and analysis algorithms to improve Collaborative Problem Solving and then enumerated requirements for future datasets to be devised. Problem solving was assumed to be performed in teams of about three, four members, which talked to each other. A dataset consists of the speech recordings of such teams. The characterization methodology was based on metrics that capture cognitive, social, and emotional activities and situations. The report presented the analysis of a large group of datasets developed for Spoken Language Understanding, a research area with some similarity to Collaborative Problem Solving. I Introduction Our goal has been to study the process of problem solving by individuals, e.g., Individual Problem Solving (IPS), and teams, i.e., Collaborative Problem Solving (CPS) [ 1 , 2 ] . A main feature of our work has been studying team dynamics during CPS, as it is expressed by speech dialog during problem solving. In addition to studying Machine Learning (ML) and decision-making algorithms that use data collected from experimental settings [ 2 ] , our work considered agent-based simulation models that mimic lab experiments [ 4 , 3 , 5 ] , modeling of the behavior of research teams [ 6 , 7 ] , and applications beyond problem solving, such as therapy [ 8 ] . This previous work has highlighted the importance of having access to datasets that comprehensively represent the characteristics of IPS and CPS, so that the datasets can be used in ML training, for model construction, and analysis. The CPS process can be analyzed along four different perspectives: (i) As a complex process that involves cognitive, social, and emotional aspects interrelated in many ways [ 9 , 10 , 12 , 13 ] , (ii) As a process decided by the features of the tackled problems, which can pertain to three categories: well-defined, ill-defined, and open-ended problems [ 15 , 16 , 17 ] , in which problems descriptions and CPS processes might incorporate constraints and requirements, like timing constraints, cost, robustness, ethical aspects, and so on [ 18 , 19 ] , (iii) As a process that creates besides a solution also other outcomes, like learning new knowledge [ 14 ] or new team behavior, i.e. how to effectively work in a team [ 21 , 20 ] , and (iv) As a context-dependent process, in which the environment in which the process is performed is essential, such as organization, community, and society [ 22 , 23 ] . Our research has focused mainly on understanding the interplay between cognitive, social, and emotional aspects during CPS. Using our research experience as a starting point, the goal of this report was to discuss the dataset requirements as well as the characteristics of the existing datasets in devising and training modern ML models, decision making methods, and analysis algorithms, so that they can improve the CPS process. This work assumed that CPS was conducted in small teams of around four members, which talk to each other while solving a problem together. A dataset represents the speech recordings of such teams. It is accepted that the quality of datasets is critical for ML training, e.g., the training and re-training of deep neural networks, like transformers and Large Language Models (LLMs). The characterization methodology is based on the cognitive, social, and emotional activities of problem solving, which the report argues should be captured and characterized to understand any CPS process. The report enumerates these activities and summarizes the CPS process. Then, the report presentation refers to Spoken Language Understanding (SLU), a research domain in which spoken language datasets were devised to automatically model individual and team activities. We felt that SLU tasks resemble CPS activities to some level. A set of CPS-based metrics were proposed and used to characterize the main SLU datasets. Then, the report discusses the needs related to dataset development to devise new ML methods to improve the CPS process. The report uses the term utterance to denote a unit of speech or text that represents a complete expression, such as a sentence, phrase, or command. It is typically defined by natural boundaries, such as pauses or shifts in conversational turns. The report has the following structure. Section II offers an overview of individual and collective problem solving. Section III summarizes SLU and compares it to CPS. Section IV offers a taxonomy of the datasets used in SLU, and the SLU activities that they address. Section V presents the quantitative characterization to study the suitability of SLU datasets to be used to represent CPS. Section VI discusses the CPS-based requirements of new datasets to be created. Conclusions end the report. II Overview of Individual and Collective Problem Solving II-A Problem Solving: Cognitive Dimension Problem-solving activities belong to the following categories [ 10 , 12 , 11 ] : 1. Problem Framing (PF) : PF requires collecting the requirements and constraints that define a problem to produce a description of the problem-solving goals [ 24 , 25 ] . Goals can refer to functionality (e.g., the nature of the input processing) and performance (i.e. the processing speed). Descriptions might include ambiguities, unknowns, and can be only partially defined. Multiple problem framings are possible for a problem, especially for ill-defined and open-ended problems [ 25 ] . Reframing might be needed if the current description is incorrect or does not lead towards a solution [ 26 , 27 ] . 2. Problem Understanding (PU) : PU is the activity of associating meaning to a problem description [ 28 ] . It involves identifying the inputs and outputs of the problem, their attributes (features), the connections between inputs and outputs, the variables involved in processing and their characteristics, the relationships between different input, output and variable connections, and the nature of the ambiguities and unknowns of a description. 3. Information Recollection (IR) : IR involves retrieving from the memory the knowledge related to solving a problem [ 31 ] . It involves information cueing from the memory and remembering similar problems, which were previously solved, and which can serve me modified or as analogies to solving the new problem [ 32 , 33 ] . More complex situations involve knowledge restructuring and sudden insight [ 34 ] . 4. Identifying the Solving Approach (ISA) : ISA represents the activity of devising the general strategy for solving a problem [ 35 , 36 ] . It produces a description, which while cannot be executed as it is not detailed enough, it serves as a starting point for solution elaboration, so that the general strategy can be converted into an executable solution. The description describes the broad operations (processing steps) required to realize the problem description. 5. Problem Decomposition (Divide and Conquer) (DC) : DC (segmentation) decomposes a problem into its subproblems and assigns subgoals to each subproblem [ 37 ] . The solution to the subproblems must be combined into the overall solution. 6. Solution Elaboration (SE) : SE details a solution description deemed to

Read the complete text