ClimateVID - Social Media Videos Analysis and Challenges Involved

Shiqi Xu, Moritz Burmester, Katharina Prasse, Isaac Bravo, Stefanie Walter, Margret Keuper · 2026

Read ClimateVID - Social Media Videos Analysis and Challenges Involved by Shiqi Xu, Moritz Burmester, Katharina Prasse, Isaac Bravo, Stefanie Walter, Margret Keuper (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27968.txt.

Introduction

Xu and colleagues study short social videos about climate change, testing whether current vision-language models can spot climate themes and how clustering reveals visual patterns. The tested models could not reliably recognise climate-specific classes, but clustering separated the material into distinct visual frames. Two image embedding models grouped the videos differently, one by style and abstract type, the other by finer detail.

Catalog record

CREATORShiqi Xu, Moritz Burmester, Katharina Prasse, Isaac Bravo, Stefanie Walter, Margret Keuper
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87855
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27968

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ClimateVID - Social Media Videos Analysis and Challenges Involved Shiqi Xu, Moritz Burmester, Katharina Prasse, Isaac Bravo, Stefanie Walter, Margret Keuper Abstract The pervasive growth of digital content, specifically short videos on social media platforms, has significantly altered how topics are discussed and understood in public discourse. In this work, we advance automated visual theme detection by assessing zero-shot and clustering capabilities on social media data. (1) We evaluated the capabilities of notable VLMs such as VideoChatGPT, PandaGPT, and VideoLLava using zero-shot image classification and compared their performance to the baseline provided by frame-wise CLIP image classification. (2) By treating clustering as a minimum cost multicut problem, we aim to uncover insightful patterns in an unsupervised manner. For both analysis strategies, we provide extensive evaluations and practical guidance to practitioners. While VLMs are currently not able to detect climate change specific classes, the clustering results are distinct visual frames. We find that both ConvNeXt V2 and DINOv2 produce meaningful clusters, with DINOv2 focusing more on style differences and abstract categories, while ConvNeXt V2 clusters differ in more fine-grained ways. Code available at https://github.com/KathPra/ClimateVID.git . 1 Introduction The explosion of digital content on social media platforms has dramatically transformed the landscape of communication and information sharing. Understanding how a topic is portrayed in the media landscape is essential for visual media research and a pre-requisite for effectively shaping public understanding and engagement [ 62 ] . In the context of climate change, the identification of visual themes and patterns is an active field of research [ 14 , 7 , 17 , 18 , 14 , 6 , 72 , 75 ] . This includes both the content of climate visuals [ 7 , 49 ] , but also the visual representation of a certain event [ 55 , 46 , 36 ] . Especially during the evolution of climate change denialism and skepticism [ 29 , 48 , 43 , 53 , 14 ] , understanding the visual discourse is key in order to effectively communicate facts and shape public understanding. Visual theme analysis allows researchers to trace how specific themes and perspectives circulate within public discourse [ 69 , 60 , 71 , 49 ] . Among the most dynamic types of content are short videos, which are widely used to convey everything from entertainment to political messaging and social activism. Especially when researching global topics, automated analysis is essential for the generation of insights. At the same time, advances in vision models open up new possibilities for automating video understanding at scale. In this work, we evaluate VLM capabilities in zero-shot image classification for VideoChatGPT [ 45 ] , PandaGPT [ 65 ] , and VideoLLava [ 39 ] and compare them to a baseline of frame-wise CLIP [ 59 ] . In our evaluations, we focus exclusively on open-source and open-weights methods. Given large deficits in VLM on social media images, we recommend frame-by-frame analysis for social media video understanding with pre-defined categories. We further discuss how to best handle meme content in social media videos. Additionally, we follow-up on work by Prasse et al. 2025 and Mooseder et al. 2023 by evaluating clustering with the goal of discovering topics in an unsupervised manner. Clustering video data instead of image data is a more challenging task given that videos have temporal redundancy, changing viewpoints, and multiple distinct scenes [ 68 , 16 ] . This makes it necessary to decide not only which frames to extract, but also how to combine them into a single, representative video-level embedding for clustering. Suboptimal choices in these steps risk over-representing redundant content or overlooking short but important moments, which limits the ability to uncover meaningful visual themes. We evaluate several frame selection strategies and find their effect to be benign. Given that CLIP embeddings have returned coarse clusters [ 58 ] , we evaluate DinoV2 [ 56 ] and ConvNeXt V2 [ 70 ] embedding spaces for this task. Based on our evaluations, we recommend using ConvNeXt V2 embeddings for video clustering. Phrasing the clustering task as a minimum cost multicut problem yields clusters of visual themes, confirming previous findings [ 58 , 2 ] . The identification of non-dominant visual themes opens the door for in-depth social science research such as [ 6 ] . Our main contributions are as follows: • Evaluation of VLMs (VideoChatGPT, PandaGPT, and VideoLLava), using zero-shot image classification in the context of social media videos. • Extension of minimum cost multicut clustering to social media videos including methodological ablations for video processing. • Practical recommendations for social media video analysis with the goal of assisting researchers and practitioners in the field. 2 Data We analysed Twitter videos in line with X’s terms of services, developer agreement, developer policy, and privacy agreement. For both methods assessed, we have created a representative subset of all available videos to generate insights in a resource-efficient manner. 2.1 Collection The videos were collected using the X Academic API in 2023, before the closure of the API for free academic access. The videos were selected based on the presence of either climatechange , climate change , or #climatechange in the tweet text. Tweets without videos are not part of this study. We have only included videos for which we have a full year of content available in order to allow year-wise comparison. 2.2 Licence According to X’s terms of services, the users retain all rights to their content submitted to X. By uploading content, users grant X a worldwide, non-exclusive, royalty-free license to share their data. Researchers are allowed to access X data through the API and conduct analysis while respecting the users’ reasonable expectation of privacy and refraining from investigating sensitive user information. Given that individuals’ perception of climate change may be related to their political affiliations and beliefs, we conduct our analysis exclusively on the videos and disregard who shared the video. Moreover, we report results either as a yearly aggregate or on the entire dataset. To make our study transparent, we share the list of tweet ids used for each task. Within this list, we indicate the year of posting and mark duplicate videos. Sharing X post ids complies with user post deletions and content removal. 2.3 Duplicate Detection The prevalence of reposts and direct re-uploads on social media platforms frequently results in numerous identical video instances, making deduplication necessary. We calculated hash values for the first and last non-black frame of each video and evaluated duplicates based thereon. The first video encountered was designated as the “original”, and subsequent videos were labeled as “duplicates”. Hashing was chosen primarily for its computational efficiency. Out of 129,955 videos, 51,505 videos were identified as duplicates (39.6%) and 78,450 unique videos (60.4%). 2.4 Classification subset We assessed the zero-shot classification efficacy on 44,927 videos out of 78,450 unique videos collected ( 57\% ). We randomly selected 12,000 videos per year (2019 - 2021), while in 2022 only 9,399 unique videos were available, and thus all were selected. Out of these 45,399 videos, 472 videos ( 0.6\% ) could not be processed by at least one model and were thus removed from the sample. 2.5 Clustering subset Clustering performance was evaluated

Read the complete text