Estimating Musical Surprisal from Audio in Autoregressive Diffusion Model Noise Spaces

2025

Read Estimating Musical Surprisal from Audio in Autoregressive Diffusion Model Noise Spaces (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05306.txt.

Introduction

Mathias Bjare, Stefan Lattner, and Gerhard Widmer model musical surprise from raw audio with diffusion based models, in a paper from 2025. Surprise, or how unexpected the next sound feels, shapes how listeners hear music. A recent transformer model estimated surprise well from its own guesses about what comes next.

The authors ask whether diffusion models, which build audio by removing noise step by step, can do the same job. Their estimates fit varied data better than the transformer on a standard loss measure. Two tasks test the idea in practice: rating how surprising single notes feel, and finding segment borders in multi track songs. The diffusion models matched or beat the transformer on both. Surprise read at unlike noise levels may track features at unlike time scales.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05306

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Estimating Musical Surprisal from Audio in Autoregressive Diffusion Model Noise Spaces Abstract Recently, the information content (IC) of predictions from a Generative Infinite-Vocabulary Transformer (GIVT) has been used to model musical expectancy and surprisal in audio. We investigate the effectiveness of such modelling using IC calculated with autoregressive diffusion models (ADMs). We empirically show that IC estimates of models based on two different diffusion ordinary differential equations (ODEs) describe diverse data better, in terms of negative log-likelihood, than a GIVT. We evaluate diffusion model IC’s effectiveness in capturing surprisal aspects by examining two tasks: (1) capturing monophonic pitch surprisal, and (2) detecting segment boundaries in multi-track audio. In both tasks, the diffusion models match or exceed the performance of a GIVT. We hypothesize that the surprisal estimated at different diffusion process noise levels corresponds to the surprisal of music and audio features present at different audio granularities. Testing our hypothesis, we find that, for appropriate noise levels, the studied musical surprisal tasks’ results improve. Code is provided on github.com/SonyCSLParis/audioic . 1 Introduction Surprisal, estimated via information content (IC) or negative log-likelihood (NLL) of an autoregressive model, has been proposed as a proxy estimator for perceived musical surprise as experienced by human listeners [ 1 , 2 , 3 , 4 ] . With suitable models, the IC of musical events correlates with human surprise perception and complexity, including tonal and rhythmic aspects [ 5 , 6 ] . Music analysis with IC enables quantitative information-theoretic hypotheses about music and music perception [ 7 ] . Furthermore, IC can serve as a conditioning signal for generative models [ 8 , 9 , 10 ] . Recently, [ 11 ] showed that surprisal modeling using IC calculated in the continuous audio latent space of Music2Latent [ 12 ] effectively models musical complexity, repetition reduction, and prediction of electroencephalogram (EEG) brain responses in human listeners. In [ 11 ] , a GIVT model [ 13 ] is used that calculates IC using the likelihood of one-step predictions that are assumed to follow Gaussian mixture model (GMM) distributions with uncorrelated dimensions. However, this assumption may limit such models’ effectiveness, given the nature of highly compressed continuous autoencoder representations, like Music2Latent. Diffusion models have become powerful tools in generative AI, achieving state-of-the-art results in multiple domains, including music. A key advantage is that they do not rely on strong assumptions about how the data is distributed. Recent work [ 14 ] shows that by formulating diffusion processes as ODEs, a diffusion model can not only generate new samples but also estimate how likely (or “probable”) any given data point is under the model. Remarkably, this likelihood estimate can be computed at different stages of the diffusion process, which correspond to varying levels of “noise” or abstraction in the data. In this paper, we study the ability of diffusion models to estimate musical surprisal. We restrict our investigations to ADMs [ 15 , 16 ] , since musical surprisal is causal in time. We experiment with diffusion models trained on multi-track audio following the popular EDM [ 17 ] and the Rectified Flow [ 18 ] processes, which differ in how they noise details. We show that these models describe diverse music data better in terms of NLL than GIVT, consistent with audio fidelity results in generation tasks [ 16 ] . We evaluate the estimated IC’s effectiveness in modelling aspects of musical surprisal in audio on two tasks that have been studied in the monophonic symbolic domain: capturing monophonic pitch surprisal, related to tonality understanding, and segment-boundary detection on multi-track audio, related to the information changes in music. We show that surprisal estimated using diffusion models captures pitch surprisal better than the GIVT model. Furthermore, we demonstrate that peaks in the surprisal function align with segment boundaries; however, additional peaks are found. Finally, we hypothesize that IC estimated at certain diffusion process noise levels can preserve the surprisal of higher-level audio features like pitch, while filtering out contributions to the IC of low-level features like timbre nuances. We support our hypothesis by showing that, for appropriate noise levels, the results of the studied musical surprisal tasks improve. 2 Related work In the symbolic music domain, musical surprisal proxied by IC has most notably been studied with the variable order Markov-model IDyOM [ 2 ] . IDyOM modeling of human melodic expectation has been validated by numerous behavioral and neural studies [ 3 , 19 , 20 , 21 , 22 , 23 ] . The model is, however, limited to monophonic symbolic music stimuli. In [ 10 ] , the authors propose an IC-based technique for estimating surprisal in polyphonic symbolic music and show the IC to correlate with tonal and rhythmic complexity using solo piano performances. In the audio domain, surprisal estimation typically relies on human-selected audio features. In [ 24 ] , the IC of a D-REX [ 25 , 26 ] model, calculated using Bayesian inference, is related to the magnetoencephalography (MEG) brain response of human participants. The Audio Oracle [ 27 ] analyzes surprisal using information rate calculated from self-similarities of audio features and identifies high surprisal at segment boundaries. Surprisal estimation using symbolic music or audio features faces two issues: the investigation is based on limited human-selected attributes, and a mismatch between what the computational model sees and what a human listener hears. Both cases potentially bias the investigation. Therefore, most similar to our approach, [ 11 ] estimates surprisal in an audio representation that preserves all features of the original audio. The authors estimate IC using the likelihood of a GIVT model [ 13 ] and show that it can predict EEG responses to sung music. However, in contrast to ADMs, the GIVT model assumes that the next-step predictions follow a particular distribution, which may limit its predictive effectiveness. Although unrelated to temporal surprisal, [ 28 ] uses the KL-divergence of a diffusion model to approximate the likelihood of 5-second monophonic music clips. This is used to reproduce the inverted U-shape relation between the total IC of music clips and listener preference presented in [ 7 ] . The model does not rely on audio features; however, it ignores causality and memory aspects of surprisal. In addition to IC, other measures of information have been proposed for the computational study of musical surprisal and expectation [ 29 , 30 ] . These, however, are impractical to calculate for continuous autoregressive models and have been limitedly validated perceptually in the literature. 3 Method 3.1 Information Content Modelling Estimation of causal IC in a discrete (symbolic music) domain can be achieved effectively with GPT-style one-step prediction modelling. In this case, IC is calculated from the prediction target’s log-likelihood according to an explicit (softmax) probability mass function with logits from a multi-layer perceptron (MLP) that takes as inputs a context state summarized by a causal Transformer model [ 31 ] . As a result, the IC measures the likelihood of specific (musical) events. In contrast, we aim to estimate the IC of continuous audio embeddings , using the compressed representations of the Music2Latent autoencoder [ 12 ] . In this continuous case, the probability mass cannot be modelled. Consequently, in [ 11

Read the complete text