From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics
Paolo Cupini, Francesco Pierri (✉) · 2026
Read From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics by Paolo Cupini, Francesco Pierri (✉) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26772.txt.
Introduction
Cupini and Pierri evaluate multimodal pipelines that annotate broadcast television news in an Italian setting. Broadcast clips mix tight audiovisual structure with strict working limits, a niche general video models rarely face. Their benchmark labels clips on setting, topic, sensitive content, and named entities. Two pipeline designs run across nine frontier models, with inputs enriched step by step by speech text, speaker splits, and metadata. Gains from raw video prove strongly model-dependent: big models use time flow well, while small ones gain little.
Catalog record
| CREATOR | Paolo Cupini, Francesco Pierri (✉) |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58388 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26772
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics Paolo Cupini, Francesco Pierri (✉) Abstract Automated semantic annotation of broadcast television content presents distinctive challenges, combining structured audiovisual composition, domain-specific editorial patterns, and strict operational constraints. While multimodal large language models (MLLMs) have demonstrated strong general-purpose video understanding capabilities, their comparative effectiveness across pipeline architectures and input configurations in broadcast-specific settings remains empirically undercharacterized. This paper presents a systematic evaluation of multimodal annotation pipelines applied to broadcast television news in the Italian setting. We construct a domain-specific benchmark of clips labeled across four semantic dimensions: visual environment classification, topic classification, sensitive content detection, and named entity recognition. Two different pipeline architectures are evaluated across nine frontier models, including Gemini 3.0 Pro, LLaMA 4 Maverick, Qwen-VL variants, and Gemma 3, under progressively enriched input strategies combining visual signals, automatic speech recognition, speaker diarization, and metadata. Experimental results demonstrate that video-based inputs do not systematically outperform configurations based on image frames. Performance gains from video input are strongly model-dependent: larger models effectively leverage temporal continuity, while smaller models show performance degradation under extended multimodal context, likely due to token overload. Beyond benchmarking, the selected pipeline is deployed on 14 full broadcast episodes (over 3,000 minutes), with minute-level annotations integrated with normalized audience measurement data provided by an Italian media company. This integration enables correlational analysis of topic-level audience sensitivity and generational engagement divergence, demonstrating the operational viability of the proposed framework for content-based audience analytics. 1 Introduction Broadcast television relies on minute-level audience measurement systems to monitor performance and demographic composition [ 12 , 13 ] . Metrics such as Average Minute Rating (AMR) provide fine-grained information about how many viewers are watching at each moment of a program. However, while these indicators quantify engagement, they do not explain why audience fluctuations occur. Connecting minute-level variations to the semantic structure of broadcast content remains largely manual and difficult to scale [ 14 , 15 ] . Recent advances in multimodal large language models (MLLMs) enable joint reasoning over visual frames, speech transcripts, and structured metadata, offering new opportunities for automated video annotation [ 16 , 17 , 18 ] . In principle, such models could support minute-level semantic labeling of broadcast programs and allow direct alignment with audience data [ 19 ] . Yet deploying multimodal systems in real broadcast environments introduces practical constraints: broadcast content is temporally structured and heterogeneous; speech recognition may introduce noise [ 20 ] ; and large-scale annotation must remain computationally sustainable. Moreover, it is not evident whether richer multimodal inputs — such as full video representations — consistently improve performance under operational latency and token constraints [ 21 , 22 ] , or whether simpler image-frames-based configurations offer a more practical trade-off [ 23 ] .This paper presents a real-world case study conducted in collaboration with a national broadcast media group. We develop and evaluate a multimodal semantic annotation framework designed to support professional TV audience analysis, and validate its deployment on over 3,000 minutes of broadcast content integrated with disaggregated audience measurement data. Two research questions guide the work: • RQ1. Do video-based pipelines consistently outperform image-based pipelines for broadcast video annotation, when jointly considering task performance and computational efficiency? • RQ2. Can semantic annotations generated by multimodal pipelines be integrated with minute-level audience data to identify demographic engagement patterns in broadcast television? Our work proceeds in two stages. First, we construct a domain-specific benchmark of 100 annotated broadcast clips and conduct a controlled evaluation of image-based and video-based pipeline configurations across nine MLLMs and four semantic tasks. The comparison explicitly considers both predictive performance and token consumption to identify a production-ready solution. Second, the selected configuration is deployed on 14 full broadcast episodes aired in Italy in 2025 (over 3,000 minutes). Minute-level semantic annotations are aligned with normalized AMR data disaggregated by age cohort, enabling fine-grained analysis of demographic engagement patterns. The results show that video-based inputs do not systematically outperform image-based configurations: performance gains are model-dependent and task-specific, with speech-derived context and metadata enrichment representing more robust performance drivers than temporal visual continuity [ 23 , 24 ] . The deployment demonstrates that semantic annotation can meaningfully contextualize audience fluctuations: while high-frequency editorial topics sustain a stable senior audience backbone, lower-frequency thematic segments generate localized engagement deviations with varying intensity across age cohorts [ 14 , 15 ] . Contributions • A domain-specific benchmark for multimodal broadcast video annotation, covering four semantic tasks under a structured taxonomy adapted to Italian prime-time television. • A systematic empirical comparison of image-based and video-based MLLM pipelines, with evidence that temporal video input does not universally improve performance and that metadata enrichment dominates transcription quality as a performance driver. • An operational deployment integrating automated semantic annotation with minute-level audience measurement data, demonstrating the applicability of multimodal video understanding to content-based demographic analytics. 2 Related Work The automated semantic analysis of broadcast television resides at the intersection of three distinct research domains: domain-specific video understanding, vision–language modeling, and content-driven audience analytics. Broadcast programming introduces distinctive challenges for automated analysis. Unlike short-form web videos, television programs exhibit stable production formats, recurring guest participation ecosystems, and a strong semantic coupling between spoken discourse and visual context [ 2 ] . Extant literature predominantly addresses these component tasks in isolation: speaker recognition benefits from the integration of face representations with audio embeddings [ 3 ] ; environment classification demonstrates that coarse scene labels provide meaningful contextual grounding; and topic classification relies primarily on Automatic Speech Recognition (ASR) transcripts, where error propagation to downstream tasks necessitates careful configuration evaluation [ 1 ] . These studies, however, optimize individual components rather than evaluating end-to-end pipelines under the pressure of heterogeneous semantic tasks and constrained computational budgets. Vision-Language Large Models (VLLMs) offer a unified reasoning framework for processing both visual and textual content [ 19 , 4 ] , and are increasingly deployed for structured video annotation and multimodal question answering. Despite their robust general-purpose capabilities, empirical evaluations highlight persistent limitations in fine-grained spatiotemporal reasoning. Specifically, merely extending the context window is insufficient without rigorous input structuring, and frame sampling strategies critically dictate downstream outcomes [ 5 ] . Furthermore, most benchmarks are calibrated on web-scale data; controlled comparative studies across various VLLMs concerning heterogeneous semantic tasks and broadcast-specific editorial structures remain scarce. The methodological challenge of systematically comparing LLM configurations has recently gained traction in applied settings: [ 10 ] emphasize the inadequacy of isolated metric evaluations when models exhibit inconsistent relative performance across different dimensions. Similarly, [ 11 ] demonstrate that LLM performance on domain-specific tasks is acutely sensitive to input structure, revealing substantial performance gaps between models that deviate from uniform scaling laws. Constructing domain-specific benchmarks is a fundamental prerequisite for this caliber of evaluation: [ 9 ] illustrate how the annotation challenges inherent in structured, real-world content diverge significantly from those in generic benchmarks, thereby informing the dataset construction methodology adopted in the present work. The third foundational area investigates the interplay between audiovisual content and audience behavior. Early methodologies relied on coarse metadata, such as programmatic schedules and broad genre labels; more contemporary research adopts signal-driven approaches at finer temporal resolutions. [ 6 ] demonstrate