LLM-based Fusion of Multi-modal Features for Commercial Memorability Prediction
Aleksandar Pramov · 2025
Read LLM-based Fusion of Multi-modal Features for Commercial Memorability Prediction by Aleksandar Pramov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22829.txt.
Introduction
Will viewers remember a commercial brand later? Aleksandar Pramov, writing alone in 2025, predicts that memorability. The work is part of a MediaEval 2025 task. His system fuses visual and text features. A Gemma-3 language model forms the backbone.
Visual features come from a vision transformer. Text features come from an embedding model. Projections merge both feature types. Low rank adaptation tunes the model cheaply. Prompts add expert based rationale about memorability. A tuned tree ensemble serves as baseline. The language model system proved more robust. It also generalized better on the final test set.
Catalog record
| CREATOR | Aleksandar Pramov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22829
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-based Fusion of Multi-modal Features for Commercial Memorability Prediction Aleksandar Pramov Abstract This paper addresses the prediction of commercial (brand) memorability as part of “Subtask 2: Commercial/Ad Memorability” within the “Memorability: Predicting movie and commercial memorability” task at the MediaEval 2025 workshop competition. We propose a multimodal fusion system with a Gemma-3 LLM backbone that integrates pre-computed visual (ViT) and textual (E5) features by multi-modal projections. The model is adapted using Low-Rank Adaptation (LoRA). A heavily-tuned ensemble of gradient boosted trees serves as a baseline. A key contribution is the use of LLM-generated rationale prompts, grounded in expert-derived aspects of memorability, to guide the fusion model. The results demonstrate that the LLM-based system exhibits greater robustness and generalization performance on the final test set, compared to the baseline. The paper’s codebase can be found at https://github.com/dsgt-arc/mediaeval-2025-memorability [orcid=https://orcid.org/0009-0005-9049-1337, [email protected], ] † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: MediaEval’25: Multimedia Evaluation Workshop, October 25–26, 2025, Dublin, Ireland and Online † † address: Georgia Institute of Technology, USA 1 Introduction Video memorability plays a central role in various applications such as marketing and advertisement, film-making, and higher education. Modeling video memorability is particularly challenging because it requires integrating multi-channel data (visual, audio, and text) to predict a latent, unobservable, subjective characteristic of the data. The “Memorability: Predicting movie and commercial memorability” task at MediaEval 2025 workshop competition undertakes to study methods that can both predict memorability, as well as shed light on the drivers behind both movie and commercial memorability Martín-Fernández et al. 2025 . This paper deals in particular with the latter question ( “Subtask 2: Commercial/Ad Memorability” ) by modeling the memorability score of commercial videos (from the financial industry) based on video and text features, as well as their brand memorability (i.e., how well a brand is remembered from a video). To that end, this year’s competition dataset includes text and pre-computed video features of 424 curated YouTube commercials from financial institutions. The provided data dictates the need to integrate multi-modal features for predictive modeling: numeric (e.g., video engagement metrics), text (e.g., video titles and subtitles), and visual embeddings. One important modeling limitation was the lack of access to the raw videos, making the usage of off-the-shelf modern multimodal LLMs difficult Martín-Fernández et al. 2025 . The approach undertaken here thus focuses on feature integration by applying both a gradient boosting model as a baseline and leveraging an LLM for the multimodal fusion of video metadata, textual embeddings, and visual features Esteban-Romero et al. 2024 . 2 Related Work Previous works in video memorability prediction often relied on sophisticated feature engineering, integrating components such as deep visual features, textual semantics, and spatio-temporal information to enhance predictive capabilities Shekhar et al. 2017 ; Martín-Fernández et al. 2024 . More recently, the field has embraced leveraging the vast world knowledge and complex reasoning abilities of Multimodal Large Language Models (MM-LLMs), such as Gemma and Qwen-VL, with fine-tuning techniques such as LoRA, for tasks like human perception analysis and long-term advertisement memorability Hu et al. 2022 ; Esteban-Romero et al. 2024 ; Martín-Fernández et al. 2025 ; Harini et al. 2025 ; Martín-Fernández et al. 2023 . 3 Approach 3.1 Data manipulation & Evaluation setup The training dataset for MediaEval’s 2025 Subtask 2 edition consists of a small sample of 339 videos for training and 85 videos for evaluation, sampled from the VIDEM Dataset Kiziltepe et al. 2025 - this was the only dataset used in our work. Moreover, multiple videos are from the same channel on very similar topics, e.g., videos on company earnings calls from different quarters within the same year. To that end, we employ two data manipulation steps on our training sample of 339 videos: First, we break down the biggest channel (‘‘Goldman Sachs”, with 23% of all the videos) into three ‘‘subchannels”, based on a k-medoids clustering algorithm on the embedded 1 1 1 https://huggingface.co/intfloat/e5-base-v2 video titles. This subgrouping creates a more balanced distribution of the channels (the largest one being 37). Second, we create a nested 5-fold, grouped (by channel name) & stratified (by 5 quantiles of the target variable) grouping of the data. The aim of this nested, stratified and grouped evaluation setup is to estimate the generalization error in a more robust way, while guarding against information leakage. We end up having 5 folds in the outer loop, each with 5 inner train/validation splits that will be used to elicit hyperparameters, early stopping and/or estimate the validation error. 2 2 2 Throughout the rest of this manuscript, ”CV SRCC” refers to the averaged test error across the 5 outer folds. The performance on the 85 held-out videos from the competition is referred to as ”Test SRCC.” 3.2 Model features For the modeling, we investigated the following provided (or derived) features as input to the models, the detailed integration of which is discussed in the next section: • Numerical metadata , provided by the organizers. • E5-base-v2 Embeddings of the subtitles (chunked and pooled), titles , and descriptions . • Pre-computed video embeddings , provided by the organizers. • Subtitle summaries of up to 1024 tokens, generated by gemma3-4b-it-qat , used both as E5-base-v2 embeddings and as text for model prompts. • Fold-aware few-shot gemma3-4b-it-qat generated (brand) memorability text rationales , based on the subtitles of the videos. The aim of the prompt was to mimic an expert system to evaluate (qualitatively) the (brand) memorability of the video, based on the subtitles only, along key characteristics such as brand integration, clarity of brand messaging, semantic richness, novelty and others. 3 3 3 The few-shot prompting here did not involve providing examples of full text of rationales, but there was fold-adaptive guidance with a few-shot examples of 2-3 (brand) memorability scores of the respective inner fold. These examples were chosen to be of those videos, that were the nearest neighbors by cosine similarity in the (pooled and averaged) embedding space of subtitles, titles and description. The generated rationales were then either used as E5-base-v2 embeddings, or in text form, to be a part of a prompt in one of the memorability predictive systems. 3.3 Modeling We investigated two modeling architectures with (brand) memorability as a univariate target variable. The first, a baseline histogram gradient boosted tree model ( HGBT ), took as input the various text and visual embeddings, as well as the numerical metadata. This approach naturally has an overwhelming number of features for a very limited sample; thus, strong PCA reduction for each of the individual text and image feature streams was employed as a pre-processing step. The hyperparameters of the HGBT models were tuned on the inner folds using Optuna to optimize for Spearman correlation, and the final model for each outer fold was evaluated on the outer test split. The second approach delves into the complexity of joint textual and visual modeling and closely follows a previously outlined approach by Esteban et al., of using instruction-tuned LLMs as multi-modal feature integrators Esteban-Romero et al. 2024 ; Liu et al. 2023 ; Verma et al. 2024 . Our implementation, which we call