Angular Distance Distribution Loss for Audio Classification
2024
Read Angular Distance Distribution Loss for Audio Classification (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00153.txt.
Description
Abstract Classification is a pivotal task in deep learning not only because of its intrinsic importance, but also for providing embeddings with desirable properties in other tasks. To optimize these properties, a wide variety of loss functions have been proposed that attempt to minimize the intra-class distance and maximize the inter-class distance in the embeddings space.
Catalog record
| YEAR | 2024 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Angular Distance Distribution Loss for Audio Classification Abstract Classification is a pivotal task in deep learning not only because of its intrinsic importance, but also for providing embeddings with desirable properties in other tasks. To optimize these properties, a wide variety of loss functions have been proposed that attempt to minimize the intra-class distance and maximize the inter-class distance in the embeddings space. In this paper we argue that, in addition to these two, eliminating hierarchies within and among classes are two other desirable properties for classification embeddings. Furthermore, we propose the Angular Distance Distribution (ADD) Loss, which aims to enhance the four previous properties jointly. For this purpose, it imposes conditions on the first and second order statistical moments of the angular distance between embeddings. Finally, we perform experiments showing that our loss function improves all four properties and, consequently, performs better than other loss functions in audio classification tasks. Index Terms — angular distance, audio classification, loss 1 Introduction Classification is one of the main tasks to be solved with machine learning. In this task, there are typically high-dimensional elements and the goal is to decide to which class of a finite set each of these elements belongs. For this purpose, most of the solutions, particularly those based on deep learning, involve obtaining intermediate representations of reduced dimension of the elements to be classified. These representations are called embeddings and they can be considered as a summary of these elements containing the information that is relevant for classification. This problem is very popular not only because of its intrinsic importance, but also because it provides a simple way to obtain embeddings compared to other methods. Embeddings are useful for a multitude of tasks such as anomaly detection [ 1 , 2 ] , biometric recognition [ 3 , 4 ] , etc. The standard loss function to solve the classification task is the cross-entropy. As a secondary result of using this loss function, the embeddings of the different classes usually end up being somewhat separated. However, it is common to impose certain conditions directly on them due to two reasons: (i) this tends to improve the performance in the classification problem by guiding more the optimization [ 5 , 6 ] ; and (ii) it may be desirable for embeddings to have certain properties when used for a specific task other than classification [ 7 , 8 ] . These conditions on embeddings are usually imposed through the loss function. Typically, a term is added to the cross-entropy or a modification is made to it. In this paper we propose a loss function that is added to cross-entropy and we call it Angular Distance Distribution Loss because it imposes conditions on the first and second order statistical moments of the angular distances between embeddings in order to organize the embeddings in the space. Specifically, this organization consists of: (i) bringing embeddings of the same class closer, (ii) moving embeddings of different class away, (iii) minimizing the variation of the distances of the embeddings of the same class, and (iv) making the embeddings of a class equal in distance to the embeddings of any class. Traditionally, only the first two have been considered in the literature. However, in section 3 we formalize all four, arguing why they are all important. In addition, we reason how they relate to the statistical moments of the distances between embeddings. Furthermore, we propose an experimental framework with different Audio Classification datasets. In these experiments, on the one hand, we verify that our embeddings satisfy the properties described in the previous paragraph, so we verify that our loss function encourages the properties to be satisfied. On the other hand, we obtain a better accuracy than other loss functions that aim to establish conditions on the embeddings. Thus, we verify that the described properties translate into better classification performance. The details of these experiments are presented in the section 4 and can be replicated using the code in https://github.com/antonioalmudevar/distance_distribution_loss 2 Related Work Audio Classification consists of identifying to which class an audio belongs [ 9 , 10 ] . In recent years it has received a lot of interest from the community [ 11 , 12 ] . Solutions to this problem typically involve an embedding extractor followed by a small classifier net which are trained by minimizing cross-entropy. In many SOTA solutions the embedding extractor has a large number of parameters, so it is common to pre-train it with a large dataset and perform finetuning for the desired dataset. Although convolutional architectures has been widespread used [ 13 , 14 , 15 ] , the most popular systems nowadays are transformer-based. These include Audio Spectrogram Transformer (AST) [ 16 ] and BEATs [ 17 ] . Loss Functions. It has been observed in different works that separating the embeddings of different classes often results in better performance in the classification task [ 18 , 19 , 5 , 20 ] . Two loss functions that stand out are Focal Loss [ 7 ] and Orthogonal Projection Loss (OPL) [ 5 ] , with which we compare our proposal. 3 Proposed Method The problem we seek to solve in this paper is that of canonical classification, which has two characteristics: (i) all errors are considered equally critical; and (ii) all elements are considered equally similar to each other within a class. This means that intra-class and inter-class hierarchies are not desirable. In fact, the standard evaluation metric is accuracy, which considers all errors and correct predictions equally relevant. The presence of these hierarchies is desirable in some scenarios, but our goal is not to solve the latter. 3.1 Classification Solution Formulation Let \mathcal{D}=\{x^{(i)},y^{(i)}\}_{i=1}^{N} be the dataset, where x^{(i)} is each input and y^{(i)} the label of x^{(i)} and is a vector containing at each position j the probability that x^{(i)} belongs to class j . The objective is to design a system that allows us to obtain a prediction of y^{(i)} which we denote \tilde{y}^{(i)} . Typical deep learning classifier solutions consists of (i) an embeddings extractor f_{\theta} , which provides the embedding as z^{(i)}=f_{\theta}(x^{(i)})\in\mathbb{R}^{k} ; and (ii) a classifier net g_{\phi} , which gives the predictions as \tilde{y}^{(i)}=g_{\phi}(z^{(i)})\in\mathbb{R}^{c} . Cross-entropy between y_{i} and \tilde{y}_{i} is used as loss function, which we call \mathcal{L}_{CE} . 3.2 Desirable Properties of Embeddings We explain below some desirable properties of embeddings for classification. In figs. 1 to 4 the dots correspond to low dimensional representations of the embeddings and different colors are used to indicate different classes. • Intra-class clustering : The embeddings of the same class are close to each other in space. This has been shown to improve performance in classification and additional tasks. Figure 1: Low (left) and high (right) Intra-class clustering • Intra-class equidistance : All the embeddings from the same class have approximately the same distance from each other. From a conceptual perspective, all the elements in a given class should be equally similar. Figure 2: Low (left) and high (right) Intra-class equidistance • Inter-class separation : Embeddings of different classes are far away from each other. This allows to take better advantage of all the space and, as a consequence, improves the