Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures

2025

Read Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02273.txt.

Introduction

Music producers need to know what audio effects were applied to each instrument inside a finished mix. Yen-Tung Yeh and co-authors presented a model for that task in 2025, called Fx-Encoder++. General audio models describe a whole mixture at once, which is not enough for jobs like automatic mixing that work instrument by instrument.

The model learns from contrasts between clips and uses an extractor that takes an instrument query, given as audio or text. That query pulls the effects features for just that instrument out of the mixture features. In retrieval and effects-matching tests across many instruments, the model beat earlier mixture-level methods. The authors frame it as filling a key gap in smart music production.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02273

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fx-Encoder++: Extracting Instrument-Wise Audio Effects Representations from Mixtures Abstract General-purpose audio representations have proven effective across diverse music information retrieval applications, yet their utility in intelligent music production remains limited by insufficient understanding of audio effects (Fx). Although previous approaches have emphasized audio effects analysis at the mixture level, this focus falls short for tasks demanding instrument-wise audio effects understanding, such as automatic mixing. In this work, we present Fx-Encoder++ , a novel model designed to extract instrument-wise audio effects representations from music mixtures. Our approach leverages a contrastive learning framework and introduces an “extractor” mechanism that, when provided with instrument queries (audio or text), transforms mixture-level audio effects embeddings into instrument-wise audio effects embeddings. We evaluated our model across retrieval and audio effects parameter matching tasks, testing its performance across a diverse range of instruments. The results demonstrate that Fx-Encoder++ outperforms previous approaches at mixture level and show a novel ability to extract effects representation instrument-wise, addressing a critical capability gap in intelligent music production systems. 1 Introduction Recent advances in deep learning have enabled significant progress in general-purpose audio representations [ 1 , 2 , 3 , 4 , 5 ] , which have proven effective across diverse applications. However, they inadequately capture the nuanced characteristics of audio effects processing [ 6 , 7 , 8 ] , since they prioritize semantic content recognition over subtle audio effects or timbral transformations. This shortcoming particularly affects specialized applications that require a precise understanding of audio effects configuration and their perceptual impact across different musical contexts. A key domain affected by this representation gap is intelligent music production, the emerging field of mixing and mastering automation aiming to develop AI systems capable of professional-quality audio engineering [ 9 ] . Applications in this domain, including automatic mixing [ 10 , 11 ] , audio effects style transfer [ 12 , 6 , 13 ] , and mixing style transfer [ 14 , 15 ] , all require specialized representations that model effects at two distinct levels: how they shape the overall sound of a complete mixture (“mixture level”) and how they transform individual instruments within that mixture (“instrument level”). Previous approaches to audio effects representation can be categorized into two groups: those analyzing effects at the entire musical mixes [ 15 , 12 ] and those examining effects on isolated single instruments [ 16 , 17 ] . Although both approaches operate at what we consider the “mixture level”, they fail to extract instrument-specific effect characteristics from complex mixtures. To our knowledge, FX-Encoder [ 15 ] is the only existing work that encodes audio effects information from individual instrument stems. However, they focus only on modeling the aggregate result, rather than identifying how effects have been applied to each instrument. This limitation restricts applications such as automatic mixing, where a precise understanding of the instrument-wise processing is essential. Extracting instrument-specific audio effects representations from mixtures presents two distinct challenges. First, a straightforward approach might involve applying source separation algorithms [ 18 , 19 ] to isolate individual instruments before analyzing their effects. This method is limited to separation artifacts, such as missing high frequencies, transient smearing [ 20 ] , and imperfect signal reconstruction. Those artifacts will distort effect characteristics, making the separate-then-analyze pipeline unreliable. Second, to obtain accurate effect parameters, we require both the processed track (wet audio) and its unprocessed counterpart (dry audio) for reverse engineering [ 21 , 22 ] . This necessitates not only perfect source separation to extract the processed track from the mixture but also access to the original unprocessed audio, which is rarely available. To address these limitations, we propose Fx-Encoder++ , a novel model designed to extract instrument-specific audio effects representations from complete music mixtures. Our model employs a contrastive learning framework based on SimCLR [ 23 ] , which learns representations by maximizing agreement between different augmented views of the same data. Following FX-Encoder [ 15 ] , we implement several crucial design elements to ensure effective learning for audio effects including: i) Fx-Normalization [ 24 ] to normalize inherent effects in source audio; ii) consistent instrumentation composition, to ensure each mixture in the entire batch is constructed with the same instrument combination, thereby isolating effect-related features; and iii) systematic audio effects manipulation procedures. The core of our contribution is an “extractor” mechanism that transforms mixture-level embeddings into instrument-specific embeddings when provided with instrument queries. This extractor leverages a pretrained CLAP encoder [ 3 ] to support both audio and text queries. In our experiments, we evaluate Fx-Encoder++ against both general-purpose audio representations (VGGish [ 1 ] , PANN [ 2 ] , and CLAP [ 3 ] ) and specialized audio effects encoders (FX-Encoder [ 15 ] and AFx-Rep [ 6 ] ). Building upon the evaluation framework of AFx-Rep [ 6 ] , we introduce a comprehensive benchmark for audio effects retrieval using real-world multitrack recordings from MUSDB [ 25 ] and MedleyDB [ 26 ] . We quantitatively assess performance across two dimensions: effects complexity (single vs. multiple cascaded effects) and instrument generalization. Results demonstrate that Fx-Encoder++ consistently outperforms existing methods in mixture-level effects extraction tasks. Furthermore, our approach enables instrument-specific effects extraction, addressing a gap in previous work where effects could only be analyzed at the mixture level. Figure 1 : Our data preparation pipeline creates training pairs by splitting source tracks ( n=2 in this example) into segments, applying consistent effect chains, and combining them into mixtures with identical effect configurations but distinct musical content. The architecture consists of an encoder that processes mixtures to produce mixture-level embeddings, and an extractor mechanism that transforms these embeddings into instrument-specific representations using CLAP-derived queries. During inference, we can optionally provide text query to extract the specific instruments. 2 Related works 2.1 General-purpose Audio Representation General-purpose audio representations have emerged to support various downstream tasks. VGGish [ 1 ] and PANN [ 2 ] employ CNNs trained on AudioSet [ 27 ] for audio classification and pattern recognition. More recently, CLAP [ 3 ] uses a transformer-based architecture with contrastive learning to align audio and text, enabling applications in text-to-music generation [ 28 , 29 ] and audio separation [ 30 , 31 ] . Neural audio compression models [ 4 , 32 , 33 ] represent another category, utilizing VAEs to reconstruct perceptual features while minimizing bitrate. Despite their success across various tasks, these representations show limited sensitivity to audio effects transformations [ 6 , 7 , 8 ] . This limitation stems from training objectives prioritizing semantic content recognition or cross-modal alignment rather than preserving the subtle timbral modifications introduced by audio effects, a capability essential for intelligent music production systems. 2.2 Audio Effects Representation Audio effects representation research has evolved from implicit to explicit approaches. Early works incorporated effects awareness indirectly through applications like reverb parameter estimation [ 34 ] , automatic mixing [ 35 ] , differentiable effects [ 36 ] , and neural style transfer [ 12 , 14 ] . Dedicated representations emerged with FX-Encoder [ 15 ] , which used contrastive learning to disentangle effects characteristics from content, enabling mixing style transfer. Similarly, Tone Embedding [ 16 ] and OpenAMP [ 17

Read the complete text