Loss functions incorporating auditory spatial perception in deep learning – a review
Boaz Rafaely, Stefan Weinzierl, Or Berebi, Fabian Brinkmann · 2025
Read Loss functions incorporating auditory spatial perception in deep learning – a review by Boaz Rafaely, Stefan Weinzierl, Or Berebi, Fabian Brinkmann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19404.txt.
Introduction
Rafaely, Weinzierl, Berebi and Brinkmann review in 2025 how loss functions in deep learning can capture the way listeners perceive spatial sound. Binaural audio aims to give headphone listeners a realistic sense of space. Standard error measures compare waveforms, not what people actually hear. The review surveys losses built on spatial cues, judging them against a quality inventory for spatial audio.
Most work targets localization cues, such as timing and level differences between the ears. Room effects like reverberation are rarely built into losses. Newer room-estimating networks hint at a fix. The authors chart where perceptually grounded training could go next.
Catalog record
| CREATOR | Boaz Rafaely, Stefan Weinzierl, Or Berebi, Fabian Brinkmann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 8 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-19404
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Loss functions incorporating auditory spatial perception in deep learning – a review
Boaz Rafaely, Stefan Weinzierl, Or Berebi, Fabian Brinkmann
Abstract
Binaural reproduction aims to deliver immersive spatial audio with high perceptual realism over headphones. Loss functions play a central role in optimizing and evaluating algorithms that generate binaural signals. However, traditional signal-related difference measures often fail to capture the perceptual properties that are essential to spatial audio quality. This review paper surveys recent loss functions that incorporate spatial perception cues relevant to binaural reproduction. It focuses on losses applied to binaural signals, which are often derived from microphone recordings or Ambisonics signals, while excluding those based on room impulse responses. Guided by the Spatial Audio Quality Inventory (SAQI), the review emphasizes perceptual dimensions related to source localization and room response, while excluding general spectral–temporal attributes. The literature survey reveals a strong focus on localization cues, such as interaural time and level differences (ITDs, ILDs), while reverberation and other room acoustic attributes remain less explored in loss function design. Recent works that estimate room acoustic parameters and develop embeddings that capture room characteristics indicate their potential for future integration into neural network training. The paper concludes by highlighting future research directions toward more perceptually grounded loss functions that better capture the listener’s spatial experience.
I
Introduction
Binaural synthesis creates immersive audio experiences through headphones by simulating how sounds reach the ears in real acoustic environments. These systems rely on accurately rendering spatial auditory cues, often using signal processing methods that incorporate head-related transfer functions (HRTFs). As binaural audio becomes increasingly important in applications such as virtual reality, gaming, hearing aids, and telepresence, and as deep learning methods become more widely adopted in audio processing, there is a growing need for suitable loss functions—metrics used to guide the training and evaluation of learning-based algorithms. In the context of binaural audio, effective loss functions must go beyond conventional signal accuracy and incorporate perceptually relevant aspects of spatial hearing, ensuring that the optimized outputs preserve realistic and spatially accurate auditory experiences.
Recent advances in spatial audio research have increasingly emphasized perceptual aspects, particularly through structured descriptors such those summarized in the Spatial Audio Quality Inventory (SAQI)
[
lindau2014SAQI
]
. This reflects a broader trend toward integrating psychoacoustic insights into the design and processing of spatial audio systems
[
brandenburg2019perceptual
,
Hacihabiboglu2017Review
]
. Several recent studies have examined perceptually relevant attributes of spatial hearing, such as localization, envelopment, externalization, and reverberation perception, and have explored how these attributes can be evaluated using auditory models
[
Llado2025
]
. Other recent efforts have reviewed the use of machine learning for HRTF individualization
[
Fantini2025
]
and proposed data-driven pipelines for spatial audio capture and reproduction
[
cobos2022Review
]
. Nevertheless, as emphasized in earlier work on spatial audio perception
[
BRADLEY2011
,
Weinzierl2018RAQI
]
, perceptual qualities related to reverberation and room acoustics play a critical role in spatial realism. Despite this, such attributes remain underrepresented in the design of current loss functions for learning-based spatial audio processing.
This paper reviews loss functions that explicitly incorporate spatial perception attributes and evaluates their relevance to binaural reproduction. The focus is on loss functions applied to binaural signals, or signals derived from microphone arrays or Ambisonics representations, as illustrated in Fig.
1
. We exclude metrics designed solely for characterizing systems, such as room impulse responses of transfer functions. Guided by the SAQI framework, the review identifies key limitations in current approaches—particularly the limited treatment of reverberation and room acoustics—and outlines opportunities for developing loss functions that are more closely aligned with perceptual spatial audio cues.
Fig. 1:
The framework representing the signal processing chain and the incorporation of loss functions. Examples of signal processing tasks and methods within this framework are presented in the recent review paper by Rafaely et al.
[
rafaely2022Review
]
, whereas the loss functions are discussed in the present paper.
II
Spatial audio qualities
The Spatial Audio Quality Inventory (SAQI,
[
lindau2014SAQI
]
) contains perceptual qualities relevant to the evaluation of audio in virtual and augmented reality. It can serve as a starting point for identifying qualities for which loss functions are desirable. The qualities in SAQI are divided into eight categories. For the scope of this article, the current review focuses on selected qualities from two categories:
geometry
, containing qualities related to the spatial perception of the sound source; and
room
, containing qualities related to the spatial perception of the acoustic environment. A large set of room acoustic qualities was further developed in the Room Acoustic Quality Inventory (RAQI,
[
Weinzierl2018RAQI
]
). While a subset of these qualities is also contained in SAQI, other high-level qualities, such as the ‘ease of listening’ and ‘liveliness’, are excluded from the current work.
In addition to the above, we included the (overall) difference as an integrative quality and summarize the eight SAQI qualities given in the
timbre
category as a single quality, termed ‘coloration’. Strictly spoken, these are not spatial qualities, but we included them because they are important qualities for evaluating spatial audio in general. Coloration, which we consider to be mostly an attribute of the sound source, has an outstanding importance to the overall quality of (spatial) audio in general
[
Rumsey2005
,
LeBagousse2012
]
. Table
lists the selected qualities, and the reader is referred to Lladó et al.
[
Llado2025
]
for a more comprehensive review of the SAQI qualities, related measures, and available auditory models.
The overview in Table
shows that a relatively small number of technical measures is connected to the selected spatial audio qualities. While these measures could (mostly) be implemented as loss functions, their exact relation to the corresponding perceptual quality is often non-trivial. Many measures associated with source-related qualities are used as building blocks of more elaborated auditory models, for example for sound source localization
[
Majdak2022
]
; however, these may not be implementable as loss functions. In other cases, the Just Noticeable Difference (JND) of the measures is not fully understood
[
BRADLEY2011
]
, which makes applying them in a perceptually meaningful manner difficult. These factors make developing loss functions tightly linked to perceptual qualities challenging.
Object
Quality
Description
Measures
\mathcal{S,R}
Difference
Existence of a noticeable difference.
All of the below
\mathcal{S}
Coloration
Loudness differences between a reference and test stimulus, integrated across ears and frequency
(Differences of) spectral cues
\mathcal{S}
Horizontal direction
Direction of a sound source in the horizontal plane
ITD, ILD
[1pt/1pt]
\mathcal{S}
Vertical direction
Direction of a sound source in the vertical plane
Spectral cues
[1pt/1pt]
\mathcal{S}
Front-back position
Position of a source in front or behind the listener
Spectral cues
[1pt/1pt]
\mathcal{S}
Width, height, depth
Perceived extent of a source in the horizontal, vertical and radial directions
ASW, IACC
[1pt/1pt]
\mathcal{S}
Distance
Perceived distance of a source
Loudness, DRR
[1pt/1pt]
\mathcal{S}
Externali- zation
Perception of a source within or outside the listener’s head