Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization

Chaoqun Cui, Liangbin Huang, Shijing Wang, Zhe Tong Zhaolong Huang, Xiao Zeng, Xiaofeng Liu · 2025

Read Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization by Chaoqun Cui, Liangbin Huang, Shijing Wang, Zhe Tong Zhaolong Huang, Xiao Zeng, Xiaofeng Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08550.txt.

Introduction

Dubbing a film is a timing puzzle: the translated line must fit the actor's mouth. Chaoqun Cui and co-authors teach a language model to prefer translations whose spoken length matches the original, segment by segment. Their method samples candidate renderings piece by piece and trains on the fine-grained preference between well-timed and ill-timed versions, instead of judging whole lines at once.

Experiments show the approach beats prior methods at duration alignment, the mismatch that most visibly breaks the illusion for viewers. The mismatch stems from languages packing information at different densities, so literal accuracy alone cannot solve it. The authors frame dubbing quality as a preference problem, aligning the model's taste with the editor's stopwatch.

Catalog record

CREATORChaoqun Cui, Liangbin Huang, Shijing Wang, Zhe Tong Zhaolong Huang, Xiao Zeng, Xiaofeng Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08550

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization Chaoqun Cui, Liangbin Huang, Shijing Wang, Zhe Tong Zhaolong Huang, Xiao Zeng, Xiaofeng Liu Abstract Video dubbing aims to translate original speech in visual media programs from the source language to the target language, relying on neural machine translation and text-to-speech technologies. Due to varying information densities across languages, target speech often mismatches the source speech duration, causing audio-video synchronization issues that significantly impact viewer experience. In this study, we approach duration alignment in LLM-based video dubbing machine translation as a preference optimization problem. We propose the Segment Supervised Preference Optimization (SSPO) method, which employs a segment-wise sampling strategy and fine-grained loss to mitigate duration mismatches between source and target lines. Experimental results demonstrate that SSPO achieves superior performance in duration alignment tasks. 1 Introduction Video dubbing involves translating the original speech from a source language to a target language in visual media programs, relying on machine learning speech language processing techniques. Typically, video dubbing systems are not end-to-end but consist of three cascaded sub-tasks Federico et al. 2020 ; Wu et al. 2023 , namely Automatic Speech Recognition (ASR) Yu and Deng 2016 ; Chen et al. 2020 , Neural Machine Translation (NMT) Vaswani 2017 ; Cheng and Cheng 2019 , and Text-to-Speech (TTS) Wang et al. 2017 ; Ren et al. 2019 ; Li et al. 2019 ; Tan et al. 2021 . ASR converts the original speech into text. When subtitles are available or can be obtained through Optical Character Recognition (OCR) Memon et al. 2020 ; Nguyen et al. 2021 , ASR can be bypassed. NMT is used to translate the source language text to the target language, after which TTS synthesizes the translated text into speech in the target language. In video dubbing systems, maintaining strict isochronous constraints between the original source speech and the synthesized target speech in terms of speech duration is crucial for ensuring synchronization with the original video footage, which is vital for preserving an immersive experience for the audience Wu et al. 2023 . However, due to varying information densities across different languages, translating from one language to another often results in a duration mismatch between the source speech and the target speech Tiedemann 2012 ; Guzmán et al. 2014 ; Lakew et al. 2019 . For instance, when translating from Chinese, a high information density language, to lower information density languages such as English or Thai, the resulting translations frequently exceed the timing notes of the original subtitles (see Table 7 for examples of subtitle format.), significantly impacting the audience’s viewing experience. If relying solely on TTS to adjust the pause and duration of words, the vast differences in information density between languages necessitate that TTS adjusts the speaking rate of each word within a wide range to match the total speech duration. This can severely impact the fluency and naturalness of the synthesized speech, leading to a dissonance in the speaking rates between adjacent lines Wu et al. 2023 . Consequently, Duration Alignment (DA) is a significant challenge that must be addressed during NMT. Recently, Large Language Models (LLMs) have been widely applied in NMT, bringing significant improvements to translation tasks Hendy et al. 2023 ; Jiao et al. 2023 ; Zhu et al. 2024 ; Zhang et al. 2024 . LLMs have also been applied to video dubbing NMT. However, relying solely on Prompt Engineering (PE) and Supervised Fine-tuning (SFT) on human-translated subtitles does not handle DA well. This is primarily because LLMs lack direct awareness of speech duration for the text, and the available human-translated subtitles used for SFT typically focus on the text itself rather than the speech duration of the lines. For DA, although generating a translation that is a few words shorter or longer may seem like a simple task, it actually requires good control over the target language. As illustrated in Table 3 , LLMs must implicitly adopt strategies such as choosing more concise phrasing, using different verb tenses, avoiding redundant adverbs and adjectives Lakew et al. 2019 , while also maintaining translation accuracy and fluency to ensure that the translation quality does not deteriorate after DA. In this study, we consider the DA task as a special preference optimization problem, termed as a localized multi-segment preference optimization problem. This framework addresses the situation where the output of LLMs consists of multiple semantically interconnected segments. The preference metric is evaluated for individual segments rather than the entire outcome (segment supervised), requiring segment-wise alignment across each segment. To tackle this issue, we propose the S egment S upervised P reference O ptimization (SSPO) method. SSPO effectively controls the duration of translations for each line at a fine-grained level while ensuring accuracy and fluency in handling DA tasks. In summary, this study contributes as follows: • We define duration consistency metrics and established an evaluation framework for DA. • We propose the SSPO method, which formulates DA as preference optimization problem. • We elucidate the theoretical foundation of SSPO and formalize the localized multi-segment preference optimization task. • Experimental results demonstrate the effectiveness and robustness of SSPO. 2 Related Work 2.1 Duration Controllable Generation Previous research on text length control includes: 1) using reward functions or models incorporating length information to guide decoding Kikuchi et al. 2016 ; Murray and Chiang 2018 ; 2) modifying model embeddings to inject length information Lakew et al. 2019 ; Takase and Okazaki 2019 ; and 3) interfering with training using length prediction metrics or models Yu et al. 2021 ; Wu et al. 2023 ; Cui et al. 2025 . However, these methods primarily target traditional sequence-to-sequence models and are unsuitable for LLMs. This is because LLMs are highly optimized through large-scale pre-training, and modifying their embeddings or interfering with training would significantly degrade overall performance Nie et al. 2024 ; Sun and Dredze 2024 . Moreover, these methods aim to generate shorter texts, whereas DA seeks to produce translations with consistent durations to the original. Additionally, subtitle dialogues are short texts with strong contextual relationships Bassnett 2013 ; Cintas and Remael 2014 ; Cui and Jia 2024 ; Cui and Jia 2025 . In summary, unlike length control tasks aiming for overall shorter texts, DA’s objective is to generate fine-grained translations for each line in LLMs’ responses, ensuring consistent (not necessarily shorter) durations for individual lines rather than the entire response . 2.2 Language Model Preference Optimization Reinforcement learning offers an effective solution for aligning LLMs with human values and controlling text generation Bender et al. 2021 ; Bommasani et al. 2021 ; Thoppilan et al. 2022 ; Taori et al. 2023 ; Chiang et al. 2023 ; Ji et al. 2023 . Reinforcement learning from human feedback (RLHF) framework has been developed, based on human feedback reward models MacGlashan et al. 2017 ; Ziegler et al. 2019 ; Stiennon et al. 2020 ; Bai et al. 2022b ; Bai et al. 2022a ; Zheng et al. 2023 . Despite RLHF’s effectiveness, its complexity, instability, and hyperparameter sensitivity remain unresolved Engstrom et al. 2020 ; Andrychowicz et al. 2021 . Recently proposed DPO Rafailov et al. 2024 simplifies the RLHF framework by eliminating the need for

Read the complete text