LUST: A Multi-Modal Framework with Hierarchical LLM-based Scoring for Learned Thematic Significance Tracking in Multimedia Content
Anderson de Lima Luiz · 2025
Read LUST: A Multi-Modal Framework with Hierarchical LLM-based Scoring for Learned Thematic Significance Tracking in Multimedia Content by Anderson de Lima Luiz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04353.txt.
Introduction
Anderson de Lima Luiz introduces LUST, a framework that scores how relevant each segment of a video is to a theme the user describes in words. The name expands to Learned User Significance Tracker. It is meant for multimedia content where significance depends on the viewer's own brief.
The pipeline reads visual cues from frames and words from the soundtrack through speech recognition. Scoring happens in two stages: a direct score judges each segment on its own content, then a contextual score adjusts it using the scores that came before. That second pass lets the system follow an evolving storyline. Output includes an annotated video with visualized scores plus detailed analysis logs.
Catalog record
| CREATOR | Anderson de Lima Luiz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-04353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LUST: A Multi-Modal Framework with Hierarchical LLM-based Scoring for Learned Thematic Significance Tracking in Multimedia Content Anderson de Lima Luiz Abstract This paper introduces the Learned User Significance Tracker (LUST), a framework designed to analyze video content and quantify the thematic relevance of its segments in relation to a user-provided textual description of significance. LUST leverages a multi-modal analytical pipeline, integrating visual cues from video frames with textual information extracted via Automatic Speech Recognition (ASR) from the audio track. The core innovation lies in a hierarchical, two-stage relevance scoring mechanism employing Large Language Models (LLMs). An initial ”direct relevance” score, S d , i S_{d,i} , assesses individual segments based on immediate visual and auditory content against the theme. This is followed by a ”contextual relevance” score, S c , i S_{c,i} , that refines the assessment by incorporating the temporal progression of preceding thematic scores, allowing the model to understand evolving narratives. The LUST framework aims to provide a nuanced, temporally-aware measure of user-defined significance, outputting an annotated video with visualized relevance scores and comprehensive analytical logs. I Introduction The exponential growth of video data [ 1 ] necessitates sophisticated automated tools for content analysis and interpretation. A key challenge is the identification of segments that align with specific, often abstract or nuanced, user-defined themes or concepts of ”significance”. Traditional methods, often reliant on low-level feature matching or simple keyword spotting [ 2 ] , may fall short in capturing the semantic depth and contextual dependencies inherent in such tasks. To address this, this paper presents the Learned User Significance Tracker (LUST), a framework engineered to automatically identify and track user-defined thematic significance within video content over time. LUST’s methodology is built upon the synergistic integration of multi-modal data processing [ 3 ] and the advanced semantic reasoning capabilities of Large Language Models (LLMs) [ 4 , 5 ] . The system’s analysis is anchored by a user-provided textual reference summary, denoted R_{sum} , which articulates the specific theme or significance vector the user wishes to track. The principal scientific contribution of LUST is its novel two-stage, LLM-driven relevance assessment architecture: 1. Direct Relevance Assessment: For discrete video segments, an LLM evaluates the immediate relevance by jointly considering a representative visual frame I_{i} (derived from F_{i} ) and the contemporaneous transcribed speech \mathcal{C}_{S,i} in relation to the user’s defined theme R_{sum} , yielding a score S_{d,i} . 2. Contextual Relevance Assessment: This stage refines the understanding of significance by prompting an LLM to consider the direct relevance score S_{d,i} of the current segment in conjunction with a historical ledger of direct relevance scores H^{\prime}_{d,i-1} from preceding segments and the current segment’s specific speech context \mathcal{C}_{S,i} . This allows LUST to model how significance evolves and is perceived within the broader temporal narrative of the video, resulting in a score S_{c,i} . This paper details the architectural components, the multi-modal processing pipeline, and the intricacies of the LLM-based scoring mechanisms that underpin the LUST framework. II The LUST Framework: Detailed Methodology The LUST framework operates through a sequential pipeline, beginning with multi-modal input processing and culminating in LLM-driven relevance scoring and output generation. Each component is designed to extract and utilize information pertinent to assessing thematic significance as defined by the user. The primary inputs to the system are the video V and the user’s thematic reference summary R_{sum} . Key configurable parameters include the visual window duration \Delta t_{w} , speech context radius \delta_{t} , and the history length N_{hist} for contextual scoring. Input Video ( V ) User Reference Summary ( R_{sum} ) Temporal Video Segmentation & Repr. Frame Extraction Repr. Frames ( I_{i} ) Audio Extraction ( A ) & Standardization (WAV) Automatic Speech Recognition ( M_{ASR} ) Timestamped Utterances ( U ) Speech Context Aggregation for Visual Windows Speech Context ( \mathcal{C}_{S,i} ) Stage 1: Direct Relevance Scoring ( M_{LLM} ) Direct Relevance Scores ( S_{d,i} ) Stage 2: Contextual Relevance Scoring ( M_{LLM} ) Contextual Relevance Scores ( S_{c,i} ) Output Generation & Visualization Annotated Video (with S_{c,i} Overlay) Analytical Log Files Frame I_{i} Speech \mathcal{C}_{S,i} R_{sum} S_{d,i} + H^{\prime}_{d,i-1} Speech \mathcal{C}_{S,i} R_{sum}^{snip} Multi-modal Preprocessing Hierarchical LLM Scoring Fig. 1: Overall workflow of the LUST framework, illustrating the pipeline from multi-modal input processing ( V,R_{sum} ) to hierarchical LLM-based relevance scoring ( S_{d,i},S_{c,i} ) and output generation. II-A Input Modalities and Preprocessing LUST processes two primary modalities from the input video V : visual and auditory. The user also provides a crucial textual input: the reference summary R_{sum} . II-A 1 User-Defined Thematic Anchor: The Reference Summary ( R_{sum} ) The reference summary, denoted R_{sum} , is a textual input provided by the user that describes the theme, concept, event, or pattern of interest they wish to track in the video. This summary serves as the semantic ground truth or query against which video segments are evaluated. Its quality and specificity directly influence the LLM’s ability to identify relevant content. For example, R_{sum} might be ”tracking moments of escalating tension followed by a resolution” or ”identifying instances of collaborative problem-solving”. II-A 2 Temporal Video Segmentation and Visual Feature Extraction The input video V , with total duration T_{vid} , is first divided into N_{w}=\lceil T_{vid}/\Delta t_{w}\rceil contiguous temporal segments, termed ”visual windows”, where \Delta t_{w} is the configurable window duration (e.g., 1.0s). For each visual window i\in\{1,\dots,N_{w}\} : • Start time: t_{i}^{start}=(i-1)\cdot\Delta t_{w} • End time: t_{i}^{end}=\min(i\cdot\Delta t_{w},T_{vid}) • Actual duration: \Delta\tau_{i}=t_{i}^{end}-t_{i}^{start} • Center time: t_{i}^{center}=t_{i}^{start}+\Delta\tau_{i}/2 The parameter \Delta t_{w} dictates the granularity of the analysis. Within each visual window i , a single ”representative frame” F_{i} is selected, typically the frame temporally closest to t_{i}^{center} . This frame F_{i}=\text{ExtractFrame}(V,t_{i}^{center}) is converted into a PIL (Pillow) Image object [ 6 ] and then encoded into a base64 data URI [ 7 ] , denoted I_{i} , for transmission to the multi-modal LLM. The number of frames initially sampled within the window is a configurable parameter determining sampling density per second. II-A 3 Audio Processing and Automatic Speech Recognition (ASR) The audio track A is demultiplexed from the video file V . A video processing utility (e.g., FFmpeg [ 8 ] ) is used to convert A into a standardized WAV format (16000 Hz, mono, 16-bit PCM signed little-endian). This standardized audio is then processed by an ASR system, M_{ASR} . LUST employs an efficient reimplementation of OpenAI’s Whisper model [ 9 , 10 ] , with a configurable model size (e.g., a medium-sized English model). The ASR module, M_{ASR} , processes A to produce a set of N_{u} time-stamped utterances: U=\{u_{j}=(t_{j}^{u,start},t_{j}^{u,end},\text{text}_{j})\mid j=1,\dots,N_{u}\} (1) where t_{j}^{u,start} and t_{j}^{u,end} are the start and end times of utterance j , and \text{text}_{j} is its transcribed content. Voice Activity Detection [ 11 ] is used to improve transcript quality. II-A 4 Speech Context Aggregation for Visual Windows To link spoken content with visual segments, LUST aggregates ASR-transcribed utterances relevant to each visual window i . The speech context \mathcal{C}_{S,i} for window i is formed by collecting utterances from U that are temporally proximal to its center time t_{i}^{center} , within a configurable radius \delta_{t} (e.g., 2.5s). Let \mathcal{J}_{i}=\{j\mid[t_{j}^{u,start},t_{j}^{u,end}]\cap[t_{i}^{center}-\delta_{t},t_{i}^{center}+\delta_{t}]\neq\emptyset\} . The speech context is then: \mathcal{C}_{S,i}=\bigoplus_{j\in\mathcal{J}_{i}}\text{format}(u_{j}) (2) where \bigoplus denotes