SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation

Dennis Fucci* ⋄, Marco Gaido ⋄, Beatrice Savoldi ⋄ · 2024

Read SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation by Dennis Fucci* ⋄, Marco Gaido ⋄, Beatrice Savoldi ⋄ (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01710.txt.

Description

A 2024 arXiv research paper by Dennis Fucci* ⋄, Marco Gaido ⋄, Beatrice Savoldi ⋄. Spurred by the demand for interpretable models, research on eXplainable AI for language technologies has experienced significant growth, with feature attribution methods emerging as a cornerstone of this progress. While prior work in NLP explored such methods for classification tasks and textual applications, explainability intersecting generation and speech is lagging, with existing techniques failing to account for the autoregressive nature of state-of-the-art models and to provide fine-grained, phonetically meaningful explanations.

Catalog record

CREATORDennis Fucci* ⋄, Marco Gaido ⋄, Beatrice Savoldi ⋄
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-01710

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation Dennis Fucci* ⋄, Marco Gaido ⋄, Beatrice Savoldi ⋄ Abstract Spurred by the demand for interpretable models, research on eXplainable AI for language technologies has experienced significant growth, with feature attribution methods emerging as a cornerstone of this progress. While prior work in NLP explored such methods for classification tasks and textual applications, explainability intersecting generation and speech is lagging, with existing techniques failing to account for the autoregressive nature of state-of-the-art models and to provide fine-grained, phonetically meaningful explanations. We address this gap by introducing S pectrogram P erturbation for E xplainable S peech-to-text Generation ( SPES ), a feature attribution technique applicable to sequence generation tasks with autoregressive models. SPES provides explanations for each predicted token based on both the input spectrogram and the previously generated tokens. Extensive evaluation on speech recognition and translation demonstrates that SPES generates explanations that are faithful and plausible to humans. 1 Introduction The recent advances of Artificial Intelligence (AI) and the emergence of foundation models ( Bommasani et al., 2021 ) have come together with concerns about their risks and impact ( Weidinger et al., 2021 ) , as well as calls for a better understanding of their inner workings ( Barredo Arrieta et al., 2020 ; Eiras et al., 2024 ) . This need has been reinforced by the demands for transparency by legal frameworks like the EU AI Act and the US National AI Initiative Act ( Panigutti et al., 2023 ; Roberts et al., 2024 ) . In response, the field of eXplainable AI (XAI) has emerged prominently, with the goal of elucidating the reasoning behind system predictions ( Doshi-Velez and Kim, 2017 ; Carvalho et al., 2019 ; Vilone and Longo, 2021 ; Pradhan et al., 2022 ) . Among the various active XAI approaches ( Ferrando et al., 2024 ) , this paper centers on feature attribution methods, which aim to identify and quantify the importance of each input feature in determining a model’s final output. Originally developed for image and text classification ( Danilevsky et al., 2020 ; Kamakshi and Krishnan, 2023 ) , feature attribution methods have been extended to other modalities—like speech ( Becker et al., 2024 ; Pastor et al., 2024 ) —and to text generation ( Sarti et al., 2023 ; Zhao et al., 2024 , et al.) . Still, XAI in the domain of speech-to-text (S2T) generative tasks is limited, with only a few preliminary works focusing on automatic speech recognition (ASR) ( Mandel, 2016 ; Trinh et al., 2018 ; Trinh and Mandel, 2021 ; Kavaki and Mandel, 2020 ; Markert et al., 2021 ; Wu et al., 2023 ; Wu et al., 2024 ) . This contrasts with the pivotal role of spoken language—arguably the most natural form of human interaction ( Munteanu et al., 2013 ) —and consequently with the importance of S2T technologies, which are now fostered by foundation models that transcribe and translate at an unprecedented scale ( Latif et al., 2023 ) . Moreveor, most existing feature attribution methods in S2T Mandel (2016) ; Trinh and Mandel (2021) ; Markert et al. (2021) ; Wu et al. (2023) ; Wu et al. (2024) are not applied to autoregressive models, which predict each token iteratively, relying on both speech input and previous output tokens. Even the only method applied to autoregressive S2T models Kavaki and Mandel (2020) assumes conditional independence between prediction steps, disregarding the influence of previously generated text on each new prediction. Finally, these methods are either incompatible with the spectrogram input format ( Markert et al., 2021 ; Wu et al., 2023 ; Wu et al., 2024 ) , which is typically used in modern S2T models, or produce explanations that fail to highlight fine-grained patterns related to acoustic characteristics of speech, such as the fundamental frequency and formants ( Mandel, 2016 ; Trinh et al., 2018 ; Kavaki and Mandel, 2020 ; Trinh and Mandel, 2021 ) . To overcome the above limitations, we propose S pectrogram P erturbation for E xplainable S peech-to-text Generation ( SPES ), the first feature attribution technique that provides token-level explanations for autoregressive S2T models by considering both the speech input and previously generated tokens. 1 1 1 The code is available at https://github.com/hlt-mt/FBK-fairseq under the Apache License 2.0. SPES employs a perturbation-based approach that adapts image segmentation to spectrograms, enabling the identification of fine-grained, meaningful patterns. Through quantitative assessments and in-depth analyses across two S2T tasks—ASR and, for the first time, speech translation (ST)—we show that SPES produces accurate, phonetically interpretable explanations aligned with human understanding. 2 Background Feature attribution methods produce a saliency score vector S=[s_{1},s_{2},\ldots,s_{d}] , where each s_{i} represents the contribution of input feature x_{i} to the prediction y ( Samek and Müller, 2019 ; Samek et al., 2021 ; Madsen et al., 2022 ) . S is typically visualized as a heatmap, or saliency map , that highlights influential input regions. Existing methods fall into four categories: gradient-based, amortized, decomposition-based, and perturbation-based, each offering distinct ways to generate S . Gradient-based methods ( Simonyan et al., 2014 ; Sundararajan et al., 2017 ; Selvaraju et al., 2019 ) use the model’s gradients with respect to input features to measure feature importance, where the gradient reflects the sensitivity of the prediction to small changes in each feature. While these methods are efficient, as they involve a single or a few backpropagation steps, they can be unstable due to gradient instabilities and sensitivity to noise ( Ancona et al., 2018 ; Samek et al., 2021 ; Dombrowski et al., 2022 ) . Amortized explanation methods ( Dabkowski and Gal, 2017 ; Yoon et al., 2019 ; Schwarzenberg et al., 2021 ; Chuang et al., 2023b ) train a global explainer to produce saliency maps in a single forward pass. Although efficient at inference time, this approach requires separate training for each model and depends on the explainer’s learning capacity, which can be influenced by biased training patterns Chuang et al. (2023a) . Decomposition-based methods ( Bach et al., 2015 ; Shrikumar et al., 2017 ) propagate predictions backward through the network, layer by layer, decomposing the prediction score into contributions from individual neurons according to predefined rules. This process continues until the score is distributed across the input features. While these methods are as efficient as gradient-based ones, their effectiveness depends on the choice of propagation rules, which can vary across models and applications, potentially limiting their generalizability ( Montavon et al., 2019 ) . Perturbation-based methods ( Zeiler and Fergus, 2014 ; Lundberg and Lee, 2017 ; Fong and Vedaldi, 2017 ) determine feature saliency by systematically altering or masking parts of the input and observing the impact on the model’s output. Although computationally intensive, especially for high-dimensional data, they provide stable and accurate explanations independent of model architecture ( Covert et al., 2021 ) . 3 Related Works Compared to other fields, feature attribution in S2T tasks has received relatively little attention, with efforts focused only on ASR, leaving other tasks like ST unexplored. Moreover, some of the proposed methods do not base their explanations on spectrograms, which is the standard input representation for

Read the complete text