From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs

Alessandro Lucca, Francesco Pierri · 2025

Read From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs by Alessandro Lucca, Francesco Pierri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19161.txt.

Introduction

Alessandro Lucca, Francesco Corso, and Francesco Pierri test whether speech recognition is ready to subtitle Italian television on its own. Subtitles drive access and engagement, yet most recognition research leans on English and short clips. The authors worked with an Italian media company on a real production system.

Four leading systems, including two Whisper variants, were scored on fifty hours of Italian programs against professional human subtitlers. None met the accuracy the industry needs for full autonomy. All four, though, proved strong as assistants that speed up human work. The study argues for a human-in-the-loop design for long-form, non-English subtitling.

Catalog record

CREATORAlessandro Lucca, Francesco Pierri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-19161

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs Alessandro Lucca, Francesco Pierri Abstract Subtitles are essential for video accessibility and audience engagement. Modern Automatic Speech Recognition (ASR) systems, built upon Encoder–Decoder neural network architectures and trained on massive amounts of data, have progressively reduced transcription errors on standard benchmark datasets. However, their performance in real-world production environments, particularly for non-English content like long-form Italian videos, remains largely unexplored. This paper presents a case study on developing a professional subtitling system for an Italian media company. To inform our system design, we evaluated four state-of-the-art ASR models (Whisper Large v2, AssemblyAI Universal, Parakeet TDT v3 0.6b, and WhisperX) on a 50-hour dataset of Italian television programs. The study highlights their strengths and limitations, benchmarking their performance against the work of professional human subtitlers. The findings indicate that, while current models cannot meet the media industry’s accuracy needs for full autonomy, they can serve as highly effective tools for enhancing human productivity. We conclude that a human-in-the-loop (HITL) approach is crucial and present the production-grade, cloud-based infrastructure we designed to support this workflow. 1 Introduction The production of accurate subtitles has become increasingly important across a wide range of video content, from social media to broadcast television 5 . High-quality subtitles not only enhance accessibility for individuals with hearing impairments but also support viewer engagement more broadly 16 . Yet subtitle creation remains a demanding task for human operators, especially in long-form television content where reliability, timing precision, and consistency are critical. On the technological side, advances in large-scale datasets and End-to-End Neural Network architectures have substantially improved Automatic Speech Recognition (ASR) performance in speech-to-text transcription tasks 9 . However, ASR research typically evaluates systems using only Word Error Rate (WER), focusing on transcription accuracy while neglecting temporal alignment—an essential aspect of subtitle production. In addition, most benchmarks target English and short audio segments, limiting their applicability to real-world subtitling workflows. These gaps are particularly relevant in industrial media contexts, where ASR systems must operate on long-form, noisy, multilingual broadcast material. To address this need, this study evaluates ASR performance on a 50-hour dataset collected from an Italian television broadcaster and documents the design and deployment of an industrial subtitling pipeline. Our goal is to analyze performance differences between leading ASR systems and to assess the level of automation that can realistically be achieved in production settings. We therefore compare four prominent models—Whisper Large V2 10 , NVIDIA Parakeet TDT 0.6b V3 14 , WhisperX 1 , and AssemblyAI Universal 12 —contrasting their accuracy, timing robustness, and overall suitability for broadcast subtitling. Our evaluation extends beyond WER to include semantic metrics (BLEURT 15 ), subtitle-oriented metrics (Subtitle Edit Rate, SubER 18 ), and entity-level accuracy (Entity Error Rate, EER). We also compute WER on one-minute segments to identify characteristics of video clips that induce higher error rates. The pipeline incorporates deterministic post-processing for subtitle readability compliance and LLM-based corrections using Gemini-2.5-Flash and Gemma3-12b-it for entity correction and punctuation restoration. This comprehensive analysis informs our approach to system combination and to designing a robust human-in-the-loop (HITL) workflow for industrial subtitling. 2 The ASR systems The current Automatic Speech Recognition (ASR) systems are all based on a End To End architecture with two main blocks, an Encoder and a Decoder 9 . The Encoder takes in input the acoustic representation of the audio, typically in log-mel features, and projects it in a very high dimensional space to capture all the information. The Decoder is responsible for the temporal alignment between the acoustic features in the input and the shorter sequence of output labels. Whisper Large v2 10 is designed with a Transformer Encoder-Decoder architecture, trained as a sequence-to-sequence system on 680k hours of weakly supervised and labelled audio data. WhisperX 1 is a variation of Whisper designed for efficient long-form transcription processing, employing an intelligent preprocessing using Voice Activity Detector (VAD) and Cut&Merge strategy, and a postprocessing with a phoneme alignment model to ensure maximum temporal alignment. AssemblyAI Universal 12 is a 600-million-parameter Conformer Encoder coupled with an RNN-T Decoder, trained on a dataset composed of 12.5M hours of unsupervised data, 188k hours of supervised data, and 1.6M hours of pseudo-labelled data. To handle long-form transcriptions, it uses a divide and conquer strategy, with intelligent segmentation of the input to allow for parallel inference. NVIDIA Parakeet TDT 0.6b v3 14 is a FastConformer Encoder paired with a TDT (Token and Duration Transducer) Decoder, a combination specifically designed to achieve a balance between transcription accuracy and computational efficiency. The model has been trained on a 660k-hour subset of the Granary dataset. 3 Related Work Assessing the quality of out-of-the-box ASR transcription is a field of research explored in various dimensions. Kuhn et al. 6 assessed the performance of multiple open-source and commercial ASR systems using a dataset composed of higher education lectures in English and German, employing the Word Error Rate as the sole evaluation metric. Their findings reveal that the quality promised by vendors is only partially confirmed in practice, with significant variability observed across systems, speakers, and languages. Similarly, Ferraro et al. 4 conducted a comparative analysis of a broad range of ASR systems across heterogeneous datasets. Their results demonstrate that ASR performance, again measured through WER, is strongly dependent on the nature and composition of the dataset under examination. In the specific domain of automatic subtitling for professional broadcasting, Davitti et al. 2 investigated the readability and quality of subtitles produced by two professional ASR systems, using a semantic metric (NER) for evaluation. Their experiments, conducted on two distinct audiovisual contents, a British talk show and a U.S. film dubbed into Italian, demonstrated that the ASR-generated subtitles consistently fell short of the 98% accuracy threshold required in the broadcasting industry. In contrast, Romero-Fresco et al. 13 evaluated the Lexi ASR system using the same NER metric and provided the first certification demonstrating that an ASR system can exceed the 98% accuracy benchmark on specific types of videos, such as single-speech videos. However, previous studies do not provide a comprehensive evaluation of multiple ASR systems within the specific context of Italian-language subtitling. Most existing work focuses on either a single model, on short and controlled audio samples, or on English-language benchmarks, thus overlooking the challenges posed by real-world audiovisual productions. This research aims to bridge these gaps by systematically benchmarking several ASR systems on a large-scale dataset of real Italian television content. The evaluation covers a wide range of program types and acoustic conditions, providing a more realistic assessment of ASR performance in professional subtitling scenarios. Several studies have investigated the use of Large Language Models as reviewers within ASR correction pipelines. Ma et al. 8 show that a zero-shot corrector (GPT-3.5) struggles to enhance transcription accuracy when limited to the 1-best hypothesis, while Fedorchenko et al. 3 report improvements in the same setting for Estonian language subtitles by using GPT-4o. Li et al. 7 demonstrate that a multilingual LLM, when fine-tuned for error correction (Qwen1.5-7b), can substantially enhance ASR output relative to a general-purpose model. Wei et al. 17 focus on LLM correction for ASR transcripts in Chinese language: their extensive experiments reveal that prompting is not effective

Read the complete text