Generative Multi-modal Feedback for Singing Voice Synthesis Evaluation

Xueyan Li, Yuxin Wang, Mengjie Jiang, Qingzi Zhu, Jiang Zhang, Zoey Kim, Yazhe Niu · 2025

Read Generative Multi-modal Feedback for Singing Voice Synthesis Evaluation by Xueyan Li, Yuxin Wang, Mengjie Jiang, Qingzi Zhu, Jiang Zhang, Zoey Kim, Yazhe Niu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02523.txt.

Introduction

Xueyan Li and six co-authors work on judging songs made by singing voice synthesis systems. Modern systems can now hold pitch and keep a steady style across a song. As output quality rises, fair and useful scoring of that output becomes more pressing.

Common reward systems give a single number, which misses traits like phrasing and expressiveness. Human labels for those traits cost a lot to collect. The team's generative multimodal feedback gives richer, multi angle comments instead. Better feedback could help builders improve synthetic singers and help listeners trust quality claims. Rich notes beat a lone score when a coach must say what to fix next in a take.

Catalog record

CREATORXueyan Li, Yuxin Wang, Mengjie Jiang, Qingzi Zhu, Jiang Zhang, Zoey Kim, Yazhe Niu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-02523

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Generative Multi-modal Feedback for Singing Voice Synthesis Evaluation Xueyan Li, Yuxin Wang, Mengjie Jiang, Qingzi Zhu, Jiang Zhang, Zoey Kim, Yazhe Niu Abstract Singing voice synthesis (SVS) has advanced significantly, enabling models to generate vocals with accurate pitch and consistent style. As these capabilities improve, the need for reliable evaluation and optimization becomes increasingly critical. However, current methods like reward systems often rely on single numerical scores, struggle to capture various dimensions such as phrasing or expressiveness, and require costly annotations, limiting interpretability and generalization. To address these issues, we propose a generative feedback (i.e., reward model) framework that provides multi-dimensional language and audio feedback for SVS assessment. Our approach leverages an audio-language model to generate text and audio critiques—covering aspects such as melody, content, and auditory quality. The model is fine-tuned on a hybrid dataset combining human music reactions and synthetic critiques from a MLLMs, enhancing diversity and linguistic richness. Quantitative experiments validate the effectiveness of the proposed dataset and training strategy, demonstrating that the framework produces musically accurate and interpretable evaluations suitable for guiding generative model improvement. The code is at https://github.com/opendilab/VocalCritic . \workshoptitle AI for Music 1 Introduction Recent advances in singing voice synthesis have led to systems capable of producing performances with impressive pitch accuracy and stylistic consistency 5 ; 14 ; 25 ; 26 . However, effectively evaluating such outputs and leveraging these assessments to improve model performance remains a significant challenge 4 . Meanwhile, existing SVS systems 5 ; 14 often miss nuanced expressiveness and high-level attributes essential for compelling vocal synthesis. Therefore, feedback mechanisms—including predefined rules 8 ; 24 and learned reward models 4 ; 12 —play a vital role in this process by providing criteria that guide iterative refinement. Many existing reward designs are based on music-theoretic rules 8 ; 9 ; 20 , incorporating explicit functions for interpretable features such as rhythm. While conceptually intuitive, these approaches often lack generalizability, struggling with unseen singers or novel styles. To address this, researchers have developed more dedicated rewards (e.g. style alignment 22 ), as well as neural networks for feature extraction 1 . Another promising direction is learning reward models in a data-driven way such as human preferences 2 ; 18 . These SVS-oriented reward models further enable the use of RL algorithms like PPO 17 to fine-tune generative models. Central to this paradigm is a reward model that produces feedback signals quantifying performance such as pitch accuracy and stylistic coherence 4 ; 12 . However, several limitations are common across reward models. First, they typically output a single numerical score, which provides an oversimplified view of singing quality 18 ; 19 . Without breaking down the evaluation into specific dimensions, such scores lack interpretability and complicate statistical analysis. This obscures the reasons behind score variations and reduces the utility of feedback for model improvement. Second, reward models require explicit definitions for criterion. Yet many qualities of singing, such as phrasing coherence, are inherently subjective and resist straightforward quantification. Although some methods use text-music alignment 21 to approach this issue, reliably capturing them through automated models remains challenging. Third, most existing reward models rely on large-scale accurately annotated data. Acquiring such labels is not only resource-intensive but also necessitates domain expertise and strict quality control—a particular difficulty in the audio domain, where inherent ambiguities can introduce noise and misguide training. Motivated by these considerations, we propose a generative reward framework that offers multi-dimension feedback for SVS evaluation. Unlike conventional scalar reward models, our approach produces language and audio critiques that assess generated singing across various aspects—such as content, style, and auditory quality. It improves interpretability, expands evaluative coverage, and enables intuitive user interaction through a language-based interface. Our model takes as input a singing audio clip and a contextual text prompt, which combines background about the music and a stylistic persona describing the critic. These inputs are processed by an audio language model, which generates diverse commentary covering dimensions like melody, creativity, and overall impression. For training, we combine two complementary data sources (Figure 1 ): (1) audio segments from human reaction videos containing real-time music reviews, and (2) singing segments paired with critiques generated by a multi-modal large language model (MLLM), ensuring standardization and systematicity in commentary style. We perform SFT on open-source audio language models 6 ; 10 , with joint supervision on both text and audio outputs to maintain multi-modal information. To evaluate the framework under realistic settings, we introduce an LLM-based benchmark incorporating music-domain knowledge to measure review quality along multiple criteria: musical accuracy, completeness, factuality, and novelty. Quantitative experiments validate the effectiveness of our dataset design, preprocessing methods, and training strategy. Through this approach, we obtain multi-modal feedback signals that can guide generative model training and support downstream tasks. 2 Method Figure 1: Overview of the data processing pipeline for constructing dual-source datasets. 2.1 Dataset Construction Our dataset is designed to support the generation of singing commentary conditioned on both audio performances and contextual metadata. Adopting a unified audio-text format, it combines two complementary sources: human reaction videos that contribute authentic and enrich real-world personal styles, and a large collection of MLLM-generated feedback that ensures standardized and systematic coverage of performance aspects. A detailed comparison is provided in Appendix A and Table 2 . Each sample in the dataset includes a 20–30 second audio segment, with some pre-processing operations to minimize source-related artifacts. The audio is paired with contextual text comprising song attributes (such as background, composer, and thematic tags) and critic persona that describe aesthetic preferences and linguistic style. This metadata enables the generation of commentaries that reflect not only the content of the singing voice but also the unique persona of the critic. The unified organization of multi-modal data ensures consistent input formatting while facilitating integration across sources and supporting cross-dataset evaluation. The inclusion of contextual guidance allows the model to produce outputs that are both musically informed and stylistically coherent. Category 1: Human Reaction Data: This category is sourced from bilibili , a major Chinese video-sharing platform, where we process human reaction videos into audio-text pairs through a pipeline in Fig. 1 . A typical reaction video involves the uploader playing a music clip, pausing, and then offering commentary. To capture them, we first apply an ASR module with timestamp to extract spoken content and its precise timing. The resulting transcripts provide authentic, stylistically diverse, and often highly personalized reaction texts. Using the timestamps, we isolate not only the speech audio corresponding to each reaction but also the music segment that was commented on. As shown in the top-right of Fig. 1 , the interval between two consecutive transcripts is treated as the music under review—trimmed to a maximum of 30 seconds. However, a challenge arises: uploaders frequently interject brief comments during playback, resulting in music contaminated with speech . To mitigate this, we use AudioSep 15 , a prompt-based source separation tool, to recover clean music from these mixed segments. Finally, we construct triplets in the form of (music, reaction text, speech). Category 2: MLLM-generated Data: To encompass broad and standard styles, we construct a second dataset comprising ten distinct genres, each represented by characteristic songs, and use a MLLM to generate corresponding singing

Read the complete text