MMTM : Tri-Modal Topic Modeling for Long-Form Video via Similarity-Gated Fusion
Ali Abusaleh, Bhuvanesh Verma, Prof. Dr. Alexander Mehler · 2026
Read MMTM : Tri-Modal Topic Modeling for Long-Form Video via Similarity-Gated Fusion by Ali Abusaleh, Bhuvanesh Verma, Prof. Dr. Alexander Mehler (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29765.txt.
Introduction
Ali Abusaleh, Bhuvanesh Verma, and Alexander Mehler build a topic pipeline for long-form video that fuses speech, sound, and sight. Their MMTM joins BERTopic clustering to speech recognition and to audio and visual embeddings through a deterministic similarity gate. On German and English broadcast news, three-way modelling cut noise and topic-switching sharply while lifting topic stability. Cluster validity rose severalfold, and word coherence rose for German but not for shorter English broadcasts. The team releases a human-checked fifty-four-hour video topic corpus.
Catalog record
| CREATOR | Ali Abusaleh, Bhuvanesh Verma, Prof. Dr. Alexander Mehler |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38675 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29765
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMTM : Tri-Modal Topic Modeling for Long-Form Video via Similarity-Gated Fusion Ali Abusaleh, Bhuvanesh Verma, Prof. Dr. Alexander Mehler Abstract We introduce MMTM , a modular pipeline for topic discovery in long-form video that integrates speech recognition, audio and visual embeddings, and BERTopic clustering through a deterministic similarity-gated fusion. Evaluated cross-lingually on German ( Tagesschau ) and English (NBC) broadcast news, joint tri-modal modeling substantially improves topic quality: noise drops from 0.27 to 0.06, transition rate from 0.70 to 0.21, and normalized entropy rises from 0.84 to 0.92, indicating more coherent and temporally stable topics. Cluster validity (Calinski–Harabasz) improves by 5–12 × \times across embedding spaces. Lexical coherence (NPMI) rises from 0.77 to 0.86 on German but is corpus-dependent and does not transfer to the shorter NBC broadcasts. We release the pipeline code and a human-validated 54-hour multimodal video topic corpus with dual-annotator visual evaluation and LLM-assisted labeling. 1 1 1 Code and data release upon acceptance. 1 Introduction Long-form video, including broadcast news and live streams, constitutes a growing share of online information Budzinski et al. 2021 . Understanding their topical structure is critical for media analysis, disinformation research, and content retrieval Park and Li 2007 ; Fu et al. 2024 . However, most topic models rely on text alone ( Zheng et al. 2014 ; Lokmanoglu and Walter 2025b ) , or at most two modalities ( González-Pizarro and Carenini 2024 ) , ignoring acoustic and visual cues humans use to parse stories and segment discourse. Visual cues such as composition, color, and symbolic action ( Zhu et al. 2013 ; Lokmanoglu and Walter 2025b ) and acoustic signatures, such as ambient soundscapes, emotional valence or tonal shifts ( Kim et al. 2009 ; Hu et al. 2014 ) , function as subtextual framing devices that clarify intent and urgency in ways that mere transcription cannot reflect. Consequently, there is a compelling need to integrate these modalities to capture the semantic context inherent in the interplay between textual, acoustic, and visual information. Figure 1: Architecture of MMTM . Visualization created using PaperBanana based on the authors’ design. We introduce MMTM (Multimodal Topic Modeling in Long-Form Video via Similarity-Gated Fusion) , a modular pipeline that bridges this gap: It extracts segment-level text, audio, and visual representations with pretrained encoders, fuses them via a deterministic similarity gate, and clusters the joint embeddings into interpretable topics using BERTopic. MMTM is modular: each processing stage, including speech transcription (Whisper), audio encoding (CLAP, Pyannote, MFCC), visual encoding (OpenCLIP, SigLIP, an OpenWebUI-based VLLM), and topic clustering (BERTopic), is independently replaceable without affecting the rest of the pipeline. It supports configurable multimodal fusion weights and both unsupervised and seed-based weakly supervised topic modeling . MMTM uses an unsupervised, inference-only setting without predefined labels, training splits, or fine-tuning. This minimizes risks of data leakage, overfitting, or train/test contamination, since every video is processed independently under identical conditions. A challenge for reproducible multimodal topic modeling is the lack of datasets with validated topic labels and aligned multimodal representations. To address this gap, we release MMTM together with a multimodal video topic dataset of roughly 54 hours of broadcast news, validated by a dual-annotator human study. The annotation toolkit, including LLM-assisted topic labeling and visual intrusion/matching tests, will be released with this dataset. We evaluate MMTM on cross-lingual German and English news corpora, comparing text-only and multimodal topic models. Our results show that joint tri-modal modeling improves topic quality: noise drops from 0.27 to 0.06, transition rate from 0.70 to 0.21, and topic entropy rises from 0.84 to 0.92. Cluster validity indices ( Calinski-Harabasz , Silhouette , Davies-Bouldin ) improve by factors of 4–12, indicating stronger cross-modal structure. Lexical coherence (NPMI) rises from 0.77 to 0.86, indicating improved semantic interpretability through multimodal evidence. Our main contributions are threefold: 1. An open-source, modular pipeline for multimodal video topic modeling that combines ASR, CLAP, and OpenCLIP embeddings through similarity-gated fusion, followed by BERTopic clustering and weak supervision; 2. A multimodal video topic dataset with human-validated annotations, including dual-annotator visual evaluation and LLM-assisted topic labeling, available here 2 2 2 upon manuscript acceptance ; 3. An evaluation demonstrating improved structural and lexical topic quality through multimodal fusion. 2 Related work Topic modeling traditionally extracts latent thematic structure from text, using methods ranging from probabilistic models Blei et al. 2003 to neural approaches. However, text-only analysis is inherently limited, as real-world information is multimodal. Latent topic interpretation can be enriched through multimodal context. Consequently, attention has shifted toward multimodal topic extraction. Blei and Jordan 2003 pioneered this shift with Corr-LDA, jointly modeling images and captions. Other early probabilistic approaches include tr-mmLDA ( Putthividhy et al. 2010 ) , which learns modality-specific topic sets and predicts one from the other via regression. Similarly, Jia et al. 2011 model visual and textual words as multinomial distributions within a multimodal document random field to learn shared cross-modal topics. Zheng et al. 2014 proposed an early neural autoregressive topic model for multimodal data, jointly modeling an image’s visual words, annotation words, and class labels. More recently, NN-based methods for topic modeling have emerged. Most rely on neural variational inference ( Miao et al. 2016 ) , typically via variational autoencoders (VAEs), as in ProdLDA ( Srivastava and Sutton 2017 ) and the Embedding-based Topic Model (ETM) ( Qiang et al. 2017 ) . These neural approaches remain largely restricted to text. Addressing this gap, BERTopic ( Grootendorst 2022 ) introduced a shift by decoupling document embedding from topic representation. This paved the way for multimodal extensions by enabling independent multi-source representations to be integrated before clustering. Following the success of BERTopic, many studies adopted embedding-based topic modeling for multimodal settings. For instance, Prakash et al. 2023 introduced PromptMTopic, an unsupervised multimodal framework that uses LLMs to jointly analyze memes’ visual and textual content. González-Pizarro and Carenini 2024 added two algorithms: Multimodal-ZeroShotTM combines image embeddings with bag-of-words reconstructions, while Multimodal Contrast uses contrastive learning to align modalities in a shared topic space. They also introduced two evaluation metrics, image-embedding coherence (IEC) and image-embedding pairwise similarity (IEPS), which we adopt in our evaluation. Extending topic modeling to video data, Lokmanoglu and Walter 2025b introduced VisTopics. It extracts keyframes from news videos, generates captions with LLMs, and applies LDA to the resulting text. While motivating our focus on news video, their model remains text-based; we instead use modality-specific encodings to learn latent features across modalities and derive a unified multimodal topic distribution. Tri-modal topic modeling over text, audio, and visual streams remains largely unexplored. Prior multimodal topic models either use image-caption pairs ( Blei and Jordan 2003 ; González-Pizarro and Carenini 2024 ) or reduce video to text via captioning before applying LDA ( Lokmanoglu and Walter 2025b ) . We therefore compare against the strongest available baseline: text-only BERTopic with the same transcripts, encoder, and clustering settings, isolating the effect of multimodal fusion. 3 Methodology & Dataset In this section, we describe the video corpus constructed for topic modeling and the MMTM architecture used to process it (see Figure 1 ). 3.1 Transcription and Temporal Alignment We extract audio at 16 kHz mono for compatibility with downstream ASR models. We utilize Whisper Radford et al. 2022 for transcription,