ITO-Master: Inference-Time Optimization for Audio Effects Modeling of Music Mastering Processors

2025

Read ITO-Master: Inference-Time Optimization for Audio Effects Modeling of Music Mastering Processors (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16889.txt.

Introduction

Junghyun Koo and five colleagues present ITO-Master, a system that copies the style of a reference music master while leaving the dials in the user's hands. Mastering shapes a track's final polish. Older style transfer tools apply one fixed treatment, which may miss what the artist wants.

Their framework adds inference-time optimization: while the system runs, it fine-tunes the reference embedding to steer the output. Users can nudge results in small steps toward their intent. The team tests both black-box and white-box models of mastering gear. Scores, listening tests, and text-based checks all showed closer style matches with the added control. The authors pitch it as adaptable mastering support.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-16889

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ITO-Master: Inference-Time Optimization for Audio Effects Modeling of Music Mastering Processors Abstract Music mastering style transfer aims to model and apply the mastering characteristics of a reference track to a target track, simulating the professional mastering process. However, existing methods apply fixed processing based on a reference track, limiting users’ ability to fine-tune the results to match their artistic intent. In this paper, we introduce the ITO-Master framework, a reference-based mastering style transfer system that integrates Inference-Time Optimization (ITO) to enable finer user control over the mastering process. By optimizing the reference embedding z ref z_{\text{ref}} during inference, our approach allows users to refine the output dynamically, making micro-level adjustments to achieve more precise mastering results. We explore both black-box and white-box methods for modeling mastering processors and demonstrate that ITO improves mastering performance across different styles. Through objective evaluation, subjective listening tests, and qualitative analysis using text-based conditioning with CLAP embeddings, we validate that ITO enhances mastering style similarity while offering increased adaptability. Our framework provides an effective and user-controllable solution for mastering style transfer, allowing users to refine their results beyond the initial style transfer. 1 Introduction Music mastering is the final step in the audio production process, ensuring professional sound quality and consistent playback across music distribution platforms. This process involves applying a series of audio effects such as equalization, compression, stereo imaging, and limiting, which collectively shape the sonic characteristics and enhance the overall quality of the audio [ 1 , 2 ] . Traditionally, mastering has required skilled engineers who carefully adjust these effects based on the track’s content and desired artistic outcome. However, with the increasing volume of music production and the demand for consistency across streaming platforms, the need for automated mastering solutions has grown substantially. In response to this demand, various automatic mastering systems have emerged [ 3 , 4 , 5 ] . However, these systems operate in an unconditioned manner, applying audio effects without direct user control. To introduce adaptability, reference-based approaches have been explored, where the processing characteristics of a reference track are applied to another [ 6 , 7 ] . These methods aim to match audio features such as dynamics, tonal balance, and stereo width, offering an alternative to fully automatic mastering. However, significant challenges remain in achieving both high-quality results and controllability. Existing reference-based approaches can be broadly categorized into black-box and white-box models. Black-box models, often based on end-to-end neural networks [ 6 ] , can effectively capture high-level audio patterns but lack transparency and interpretability, making it difficult for users to modify specific aspects of the processing. In contrast, white-box models leverage either feature matching algorithms [ 7 ] or differentiable audio processors [ 5 ] to provide greater control over individual parameters. While white-box methods offer a structured and interpretable approach [ 8 ] , they are often constrained by the simplicity of their differentiable processors, which may not fully replicate the complex tools used in professional mastering. In this paper, we introduce the ITO-Master framework, a reference-based audio effects modeling of music mastering processors that incorporates Inference-Time Optimization (ITO) for finer user control. While previous style transfer methods apply fixed processing based on a reference, our approach allows users to dynamically refine the output when the initial result does not fully align with their preferences. By optimizing the reference embedding z_{\text{ref}} during inference, ITO-Master enables micro-level adjustments, allowing for more precise and targeted mastering refinements. The key contributions of this work include: (1) ITO-Master framework : A novel approach for reference-based audio effects modeling of mastering processors using ITO. (2) Comparison of black-box and white-box models : A systematic study of two paradigms for evaluating their effectiveness and trade-offs. (3) Realistic mastering processor chain : Implementation of a structured differentiable mastering pipeline to enhance the realism of white-box processing. (4) Comprehensive evaluation : Performance validation via objective metrics, listening tests, and qualitative analysis using text-based conditioning with CLAP embeddings. (a) Training pipeline of Mastering Style Converter \Psi . During this phase, Reference Encoder \Phi is trained using diverse mastering styles generated by random FX manipulation f . The target signal y is synthesized by applying the same manipulation to segment A as to reference segment B , both from the same song. (b) ITO is performed using an auxiliary (content-independent) objective function, allowing any reference music. Users can optimize z_{\text{ref}} based on their preferences for both the reference and objective function. Figure 1 : Overall pipeline of ITO-Master. 2 Related Works 2.1 Audio Effects Style Transfer Audio effects style transfer has become a significant area of research in automating and enhancing music production. Recent advancements in deep learning have led to more sophisticated approaches, where neural networks are used to learn complex mappings between input and output audio signals. These methods have been applied to style transfer across single audio effects (Fx) or multiple sets of Fx (Fx chain) [ 5 , 6 , 9 , 10 , 11 , 12 , 13 , 14 ] , effectively modeling temporal dependencies and applying style transfer based on a reference track at the waveform level. While these methods have shown success in controlled environments, challenges remain in extending their application to diverse real-world scenarios, particularly in adapting to different mastering styles and varying input signals. 2.2 Inference-Time Optimization Recently, [ 15 , 16 ] have explored inference-time optimization (ITO) in music generation tasks, where the initial latent embedding is optimized by backpropagating through diffusion-based models with the loss between a generated sample and a reference track. In the context of audio effects style transfer, ITO has been applied in methods like ST-ITO [ 17 , 18 ] , where interpretable parameters in a white-box differentiable Fx chain are optimized. Our work focuses specifically on mastering style transfer, where handling heavily compressed audio—common in commercially released music—requires particular attention to limiters and dynamic range management. The ITO-Master framework introduces ITO on the reference embedding z_{\text{ref}} , allowing optimization at inference time in both black-box and white-box models. By fine-tuning z_{\text{ref}} , our approach adapts the mastering style of a reference track without retraining the entire model. ITO-Master ensures smooth adaptation to the reference track’s characteristics while preserving the ability to observe and adjust the underlying parameters in the white-box model. This makes our framework particularly suitable for mastering tasks, providing professionals and amateurs with precise control over the audio mastering process. 3 Methodology In this section, we describe the components of the proposed mastering style transfer framework: the training pipeline, Mastering Style Converter, differentiable mastering chain, and ITO. The training pipeline simulates real-world mastering scenarios by applying random Fx manipulations. The Mastering Style Converter \Psi transfers the mastering style from a reference track to a target track and can be implemented using both black-box and white-box approaches. The differentiable mastering chain serves as a white-box processor that models various Fx in a structured sequence. Lastly, the ITO process optimizes z_{\text{ref}} at inference time to enhance style transfer performance. The following subsections describe each component in detail. 3.1 Training Pipeline of Mastering Style Transfer As shown in Figure 1(a) , the training pipeline follows established methodologies in style transfer, utilizing a self-supervised training framework with

Read the complete text