VietLyrics: A Large-Scale Dataset and Models for Vietnamese Automatic Lyrics Transcription

\authorblockN Nguyen Quoc Anh \authorrefmark 1 and Bernard Cheng \authorrefmark 2 and Kelvin Soh \authorrefmark 3 \authorblockA \authorrefmark 1 National University of Singapore, Singapore E-mail: [email protected] Tel/Fax: +65-80706568 \authorblockA \authorrefmark 2 National University of Singapor · 2025

Read VietLyrics: A Large-Scale Dataset and Models for Vietnamese Automatic Lyrics Transcription by \authorblockN Nguyen Quoc Anh \authorrefmark 1 and Bernard Cheng \authorrefmark 2 and Kelvin Soh \authorrefmark 3 \authorblockA \authorrefmark 1 National University of Singapore, Singapore E-mail: [email protected] Tel/Fax: +65-80706568 \authorblockA \authorrefmark 2 National University of Singapor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22295.txt.

Introduction

Quoc Anh Nguyen, Bernard Cheng, and Kelvin Soh give Vietnamese songs the transcription dataset they have been missing. Automatic lyrics transcription is hard anywhere, but Vietnamese adds a tonal language, where pitch changes word meaning, plus strong dialect variation, and with no dedicated corpus the task had barely been attempted.

The team curated VietLyrics, six hundred forty-seven hours of songs with lyrics aligned line by line and metadata attached, and released it publicly with their models. Off-the-shelf speech recognizers stumbled badly on the material, making frequent errors and hallucinating words during instrumental stretches. Fine-tuning the Whisper family on the new data beat existing multilingual lyrics systems, including the specialized LyricWhiz. The authors pitch the resource as a foundation for Vietnamese music computing and a template for other low-resource singing traditions.

Catalog record

CREATOR\authorblockN Nguyen Quoc Anh \authorrefmark 1 and Bernard Cheng \authorrefmark 2 and Kelvin Soh \authorrefmark 3 \authorblockA \authorrefmark 1 National University of Singapore, Singapore E-mail: [email protected] Tel/Fax: +65-80706568 \authorblockA \authorrefmark 2 National University of Singapor
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22295

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VietLyrics: A Large-Scale Dataset and Models for Vietnamese Automatic Lyrics Transcription \authorblockN Nguyen Quoc Anh \authorrefmark 1 and Bernard Cheng \authorrefmark 2 and Kelvin Soh \authorrefmark 3 \authorblockA \authorrefmark 1 National University of Singapore, Singapore E-mail: [email protected] Tel/Fax: +65-80706568 \authorblockA \authorrefmark 2 National University of Singapore, Singapore E-mail: [email protected] Tel/Fax: +65-98557620 \authorblockA \authorrefmark 3 National University of Singapore, Singapore E-mail: [email protected] Tel/Fax: +65-91186401 Abstract Automatic Lyrics Transcription (ALT) for Vietnamese music presents unique challenges due to its tonal complexity and dialectal variations, but remains largely unexplored due to the lack of a dedicated dataset. Therefore, we curated the first large-scale Vietnamese ALT dataset (VietLyrics), comprising 647 hours of songs with line-level aligned lyrics and metadata to address these issues. Our evaluation of current ASR-based approaches reveal significant limitations, including frequent transcription errors and hallucinations in non-vocal segments. To improve performance, we fine-tuned Whisper models on the VietLyrics dataset, achieving superior results compared to existing multilingual ALT systems, including LyricWhiz. We publicly release VietLyrics and our models, aiming to advance Vietnamese music computing research while demonstrating the potential of this approach for ALT in low-resource language and music. 1 1 footnotetext: © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. This is the author’s accepted version of the paper: Nguyen Quoc Anh, Bernard Cheng, and Kelvin Soh, “VietLyrics: A Large-Scale Dataset and Models for Vietnamese Automatic Lyrics Transcription,” 2025 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2025. The final version is available on IEEE Xplore via DOI: https://doi.org/10.xxxx/APSIPAASC.2025.xxxxx. 1 Introduction In the modern music industry, Automatic Lyrics Transcription (ALT) has become increasingly relevant and important. It describes the process of leveraging deep learning models to convert vocal recordings into written text, making song lyrics accessible and searchable. This enhances the user experience on streaming platforms, aids in music analysis, and supports subtitle creation for music videos. It also promotes inclusivity for the hearing impaired and non-native speakers. As the industry evolves, accurate lyrics transcription is essential for greater engagement and accessibility. Despite advances in Western music computing research, Vietnamese music research lags behind, especially lyrics transcription, which remains mostly unexplored due to the lack of a large-scale, high-quality Vietnamese ALT dataset. Without technology like ALT or auto-tagging, Vietnamese media companies and streaming platforms lack the necessary tools to analyze, classify, and describe their content. This makes it difficult to build standardized datasets, hindering further research and innovation in the field, ultimately leaving Vietnamese music computing research stagnant. This paper aims at tackling the task of automatic lyric translation for the Vietnamese language. Accurate transcription of Vietnamese is arguably more difficult than English due to additional language characteristics, such as changes in tonality leading to different meanings. Vietnamese also contains a multitude of regional dialects, each with unique pronunciation patterns and differences in vocabulary. Furthermore, changes in the speaker’s pitch, accent, and emotionality can affect tonality, altering the diacritic mark, and thus the meaning of the transcribed word. For example, Tằm may sound similar to Tắm during singing, but they have completely different meanings. These complexities pose significant challenges in accurately transcribing Vietnamese vocals, especially when it is intertwined with melodies on the same track. In this work, these challenges are addressed by making the following key contributions: • The first large-scale Vietnamese ALT dataset, VietLyrics was constructed, which comprises 647 hours of songs with line-level aligned lyrics, supplemented with AI-predicted metadata such as gender and genre. The dataset is publicly released for research purposes, in compliance with Vietnamese copyright law. • The Whisper [ 1 ] architecture was further fine-tuned using the VietLyrics dataset, significantly improving performance over existing multilingual ALT/ASR systems. Three model variants of different sizes are released to support further research and development in Vietnamese lyrics transcription. 2 Related Works The following sections outline the current challenges in Vietnamese ALT and discuss the state-of-the-art solution (SOTA), detailed in Section 2.2 . TABLE 1: Open-source datasets Dataset Languages Songs (Hours) DALI-full [ 2 ] 30 (mostly EN) 5,358 DALI-train [ 2 ] 1 (EN) 3,913 MulJam [ 3 ] 6 (EN, FR, DE, ES, IT, RU) 6,031 (381.9h) JamendoLyrics [ 4 ] 4 (EN, ES, DE, FR) 80 DSing30 [ 5 ] 1 (EN) 4,324 (149.1h) MUSDB18 [ 6 ] 1 (EN) 150 (10h) VietLyrics-full (Ours) 1 (VN) 8,440 (647.1h) TABLE 2: Qualitative Analysis on Baseline LyricsWhiz’s prediction Ground Truth Prediction Source Song Analysis (No Vocals) Thành thành chuyện Thành thành; Hãy subscribe cho kênh Ghiền Mì Gõ Để không bỏ lỡ những video hấp dẫn;… Ừ Thì Anh Sai - Thiên Tú Long stretches of complete hallucination during no-vocals, due to the ASR model is not addapted into ALT (No Vocals) Lời Tắt Này Lời Tắt Này; What the hell;… 365 Ngày Yêu (Remix) - Lương Gia Huy Insertion of hallucinative random phrases Nước mắt em rơi cho cõi lòng em nắm tan Ngược mắt em rơi cho cõi lòng em đã tan Anh Biết Không Anh - Lưu Ánh Loan Substitutions of single words that can reasonably be misheard Chẳng lẽ ta mất nhau để em đây khổ sầu ; Thì ta đã lìa xa;… Chẳng lẽ ta mất nhau để em đây khô sâu ; Thì ta đã lịa xa;… Anh Biết Không Anh - Lưu Ánh Loan Substitutions or deletion of a single word’s diacritic that can reasonably be mis-transcribed due to regional accents. 2.1 Dataset Scarcity Table 1 presents a non-exhaustive list of open-source song-lyric datasets. There is a lack of large-scale non-English song datasets in general, due to copyright law. These datasets circumvent this issue by collecting music exclusively under Creative Commons licenses or in the Public Domain [ 2 , 4 , 6 ] , leveraging AI-generated lyrics [ 3 ] , and incorporating karaoke vocals voluntarily shared by app users [ 5 ] . None of these include Vietnamese, based on our findings, which leads to the lack of Vietnamese ALT models. 2.2 Limitations in Existing Solutions Although open source and commercial solutions are available for ALT or Automatic Speech Recognition (ASR) tasks, the general performance on Vietnamese music is error-prone. Various solutions were tested, including Whisper, LyricWhiz, PhoWhisper and one closed-source commercial API. While all solutions resulted in some degree of substitution, insertion, deletion errors, along with long stretches of hallucination during periods with only melody and no vocals, ASR application Whisper stands out because it wrongly recognizes the audio as Thai or other languages regularly. Among existing ALT frameworks, LyricWhiz [ 3 ] is the current SOTA for multilingual ALT, leveraging both Whisper for transcription and GPT-4 for refining outputs. Whisper serves as the “ear” by converting audio into text, while GPT-4 functions as the “brain”, selecting the most accurate lyrics. While LyricWhiz has demonstrated strong performance across various prominent languages, its effectiveness on Vietnamese lyrics is limited. The replicated baseline model struggles with Vietnamese audio, as shown in the qualitative results in

Read the complete text