Exploring Vision Language Models for Multimodal and Multilingual Stance Detection
Jake Vasilakes, Carolina Scarton, Zhixue Zhao · 2025
Read Exploring Vision Language Models for Multimodal and Multilingual Stance Detection by Jake Vasilakes, Carolina Scarton, Zhixue Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17654.txt.
Introduction
Jake Vasilakes, Carolina Scarton, and Zhixue Zhao tested vision-language models on stance detection across languages and media in a 2025 paper. Stance detection judges whether a post supports or opposes a target. Most past work used text alone, while real social media mixes images and words in many languages.
The authors extended a data set to seven languages with combined image and text posts. They probed how the models use visual cues and whether habits change by language. The models leaned far more on text than images everywhere. Within images, written words mattered most. Consistent text in the poster's stance suggests translation-focused tools may carry much of the load.
Catalog record
| CREATOR | Jake Vasilakes, Carolina Scarton, Zhixue Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17654
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Vision Language Models for Multimodal and Multilingual Stance Detection Jake Vasilakes, Carolina Scarton, Zhixue Zhao Abstract Social media’s global reach amplifies the spread of information, highlighting the need for robust Natural Language Processing tasks, like stance detection, across languages and modalities. Prior research predominantly focuses on text-only inputs, leaving multimodal scenarios, such as those involving both images and text, relatively underexplored. Meanwhile, the prevalence of multimodal posts has increased significantly in recent years. Although state-of-the-art Vision-Language Models (VLMs) show promise, their performance on multimodal and multilingual stance detection tasks remains largely unexamined. This paper evaluates state-of-the-art VLMs on a newly extended dataset covering seven languages and multimodal inputs, investigating their use of visual cues, language-specific performance, and cross-modality interactions. Our results show that VLMs generally rely more on text than images for stance detection and this trend persists across languages. Additionally, VLMs rely significantly more on text contained within the images than other visual content. Regarding multilinguality, the models studied tend to generate consistent predictions across languages whether they are explicitly multilingual or not, although there are outliers that are incongruous with macro F1, language support, and model size. 1 1 1 We will make our code and data publicly available upon publication. Introduction The ubiquity of social media has led to an increased reliance on these platforms for news and science communication, often surpassing traditional, slower-to-publish sources such as print news media and peer-reviewed scientific journals ( Matsa and Eva 2015 ; Gürer, Hubbard, and Bohon 2023 ) . A side-effect of this is an increase in the speed of information spread, including rumors and fake news ( Ceylan, Anderson, and Wood 2023 ) . Understanding the spread of information on social media requires knowing the viewpoints users take with respect to claims, topics, or entities. This is the purview of stance detection, a key task within Natural Language Processing (NLP), which aims to automatically classify an author’s viewpoint with respect to a specific target. While stance detection has been studied extensively on English text, efforts to extend it to modalities beyond text and languages beyond English are relatively sparse (cf. Küçük and Can 2020 sections 9.1 and 9.2). Additionally, the studies that do go beyond tend to investigate only one or the other, and in the case of other languages tend to focus on one or a few languages at a time. One potential reason is that obtaining high quality data requires significant time and resources. Still, extending stance detection systems to other modalities and languages is crucial, as rumors on social media often spread across languages ( Singh, Bontcheva, and Scarton 2024 ) , and posts often make use of images or videos to reinforce their message. For example, a user might comment on a news segment, use an image as supporting evidence for a scientific claim, or post a meme that expresses their view of a political figure. Recent development of Vision Language Models (VLMs), available within easy-to-use tools such as ChatGPT 2 2 2 https://openai.com/index/dall-e-3 and Huggingface 3 3 3 https://huggingface.co , provides powerful and accessible means to perform stance detection using both text and images. Furthermore, the Large Language Models (LLMs) that form the backbone of VLMs are often capable of processing and analyzing text in a variety of languages. However, to our knowledge, there has been no study investigating the ability of VLMs to perform stance detection using both text and image modalities, nor has there been any examination of their performance on this task across languages. Given this research gap, this paper investigates the performance of state-of-the-art VLMs for stance detection at the intersection of modalities beyond text and languages beyond English. Our specific contributions are threefold. • Image Use in Multimodality : an analysis of the extent to which VLMs can effectively use information from images when performing stance detection. • Multilinguality : an investigation into the performance of VLMs on stance detection across languages. • Joint Multimodality and Multilinguality : an exploration of the interaction between text and images across languages in VLMs for stance detection. Our experiments make use of a recently developed multimodal stance detection dataset in English, which we extend to cover six additional languages ( Liang et al. 2024 ) . Background We begin with an overview of the stance detection task, highlighting previous efforts on multimodal and multilingual approaches, including a discussion of currently available datasets. We then provide a review of the general architecture of VLMs, as well as previous research that attempts to use them for stance detection. Stance Detection Stance detection is the task of automatically determining the viewpoint, position, or sentiment of an author regarding a target. This target could be a topic (e.g., a news event or scientific theory), an entity (e.g., a political figure), or even the viewpoint of another author (e.g., another tweet expressing a viewpoint about a political figure) ( Küçük and Can 2020 ) . Generally, stance detection is formulated as a paired text classification problem where the input is a stance text and a target text (e.g., a tweet and the name of a political figure) and the output is one of Favour, Against , or Neutral . Stance detection is fundamental to wrangling the spread of rumors online. Interest in this task has therefore grown alongside the increased popularity of social media, which has both lowered the barrier to content publishing and increased the speed at which content can be disseminated ( ALDayel and Magdy 2021 ) . Early efforts focused on supervised classification with SVMs ( Elfardy and Diab 2016 ; Mohammad, Sobhani, and Kiritchenko 2017 ) , naive Bayes ( Walker et al. 2012 ) , or neural networks ( Siddiqua, Chy, and Aono 2019 ; Li and Caragea 2019 ) , which led to models based on pretrained transformers ( Fajcik, Smrz, and Burget 2019 ; Kawintiranon and Singh 2021 ; Khandelwal 2021 ) . The recent development of LLMs has introduced the possibility of 0-shot stance detection, although research so far has evaluated mostly GPT models ( Lan et al. 2024 ; Liyanage, Gokani, and Mago 2023 ; Suppa et al. 2024 ) . The exception is Cruickshank and Ng 2023 who performed evaluations with both 0-shot and fine-tuned LLMs, finding that fine-tuning does not necessarily increase performance. Research on multimodal and multilingual stance detection is more sparse. There are only a few studies using images alongside text ( Hu et al. 2023 ; Wang et al. 2024a ; Niu et al. 2024 ) , including Liang et al. 2024 which introduced the dataset used in this work. Stance detection on languages besides English has generally focused on one or a few languages at a time. These include Zotova et al. 2020 (Catalan and Spanish), Vamvas and Sennrich 2020 (German, French, Italian), Alhindi et al. 2021 (Arabic), and Zheng et al. 2022 (Hindi, Arabic). To our knowledge the only exception is the COFe dataset, which contains comments from a debate platform covering 26 languages ( Barriere, Jacquet, and Hemamou 2022 ) . Vision Language Models Vision Language Models (VLMs) are an extension of LLMs to images and videos. Generally, they are comprised of a vision model and an LLM.