On VLMs for Diverse Tasks in Multimodal Meme Classification

Deepesh Gavit, Debajyoti Mazumder, Samiran Das, Jasabanta Patro · 2025

Read On VLMs for Diverse Tasks in Multimodal Meme Classification by Deepesh Gavit, Debajyoti Mazumder, Samiran Das, Jasabanta Patro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20937.txt.

Introduction

Deepesh Gavit, Debajyoti Mazumder, Samiran Das, and Jasabanta Patro published this paper in 2025. It tests how well vision-language models sort memes. Memes mix image and text, and the team set the models three tasks: spotting sarcasm, flagging offense, and judging sentiment.

They tried varied prompting styles for each task and fine-tuned parts of the models to see where gains hide. Their best trick flips the usual flow. First, a big vision-language model writes a rich reading of the meme. Then a smaller language model, trained on those readings, does the sorting. That pairing beat the plain baseline by wide margins, above all on sentiment, where scores rose by about twenty-six percent. The paper maps where these models shine on memes and where they still stumble.

Catalog record

CREATORDeepesh Gavit, Debajyoti Mazumder, Samiran Das, Jasabanta Patro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On VLMs for Diverse Tasks in Multimodal Meme Classification Deepesh Gavit, Debajyoti Mazumder, Samiran Das, Jasabanta Patro Abstract In this paper, we present a comprehensive and systematic analysis of vision-language models (VLMs) for disparate meme classification tasks. We introduced a novel approach that generates a VLM-based understanding of meme images and fine-tunes the LLMs on textual understanding of the embedded meme text for improving the performance. Our contributions are threefold: (1) Benchmarking VLMs with diverse prompting strategies purposely to each sub-task; (2) Evaluating LoRA fine-tuning across all VLM components to assess performance gains; and (3) Proposing a novel approach where detailed meme interpretations generated by VLMs are used to train smaller language models (LLMs), significantly improving classification. The strategy of combining VLMs with LLMs improved the baseline performance by 8.34%, 3.52% and 26.24% for sarcasm, offensive and sentiment classification, respectively. Our results reveal the strengths and limitations of VLMs and present a novel strategy for meme understanding. 1 Introduction Multi-modal memes have gained prominence Petrova (2021) due to their eloquent and powerful way to convey complex, subtle messages Das (2023) . The widespread use of memes propagating hatred Gelber and McNamara (2016) , abuse, misogyny, and propaganda is a serious cause of concern Bhattacharya (2019) . AI models can safeguard digital platforms Hee et al. (2024) by comprehending memes. The NLP community has been trying to address this issue for years. This is evident from several publications and datasets published in recent years (See Details in Appendix (Section A )). These prevalent works introduced diverse datasets, presented efficient multi-modal deep learning (DL) models, and explored the capabilities of LLMs and VLMs Afridi et al. (2021) . We mentioned the details of these works in the Appendix. While the DL models used for the task are not generalizable Shah et al. (2024) , LLMs are incapable of processing visual information Cai et al. (2025) . The performance of the VLMs used in this task depends on their contextual understanding Xing et al. (2024) . Therefore, the existing methods are yet to attain satisfactory high efficacy necessary for the real-world deployment of these models. Since the image and the text in the memes are either complementary or indirectly linked, both modalities are essential for the task Zhong and Baghel (2024) . Besides, accurate meme understanding necessitates relating the information to the contextual understanding and the cultural backdrop Yus (2019) . Due to these predicaments, accurate meme classification is a challenging task for the AI models Jha et al. (2024) . We utilized widely used open-sourced VLMs in different settings as illustrated in Fig. 1 to address the following research questions- • R1: Are the pre-trained VLMs with diverse types of prompting powerful enough to understand memes? • R2: Does fine-tuning VLMs using adapters improve the performance? • R3: Can we combine pre-trained VLMs contextual understanding to train the LLMs for better classification? The important contributions of the work are: • Benchmarking VLM Meme Classification: We benchmark the performance of various VLMs using diverse prompting strategies. • Evaluating LoRA Fine-Tuning: We systematically explore diverse prompting strategies such as Zero Shot (ZS), Zero Shot Chain-of-Thought (ZSC), Few Shot (FS), Few Shot Chain-of-Thought (FSC), and parameter-efficient LoRA fine-tuning approach. • Novel Meme Understanding Approach: We introduce an innovative approach called Combining VLM Explanation to Fine-tune LLMs (CoVExFiL). In this approach, we use VLM-generated meme interpretations from multiple prompting techniques to train LLMs. The proposed approach with a three-step CoT prompting produces the best results. Some of the key findings from this paper are as follows- • Fine-tuning VLMs using LoRA adapter was relatively less effective for our task, because it only updates a limited number of parameters and hence do not learn the cultural context. On the other hand, the Qwen model surpassed the baseline performance for sentiment and sarcasm classification under FS prompting by 16.71%, and 0.39%, respectively. • The proposed CoVExFiL strategy yields noticeable performance gains in tasks such as SR and OF, with a significant 26.10% improvement in SN, outperforming SOTA. This is attributed to its strong meme understanding, and error analysis further confirms the effectiveness of training on generated explanations. Figure 1: Schematic diagram of the three strategies mentioned in the experiment section. 2 Datasets and Experimental Setup We used two popular datasets, i.e., Memotion by Sharma et al. (2020) and multimedia automatic misogyny identification (MAMI) by Fersini et al. (2022) in our experiments. While memotion categorizes the memes in five emotion categories (humour (HM), sarcasm (SR), offensiveness (OF), sentiment (SN), and motivational (MV)), MAMI labels whether a meme is misogynistic (MG) and categorizes the type of misogyny (as detailed in Appendix B.1 ). We use open-source vision-language models; LLaVA (LV), Qwen (QW), LLaMA (LM), and InstructBLIP (IB) for our experiments. For context, we also include results from deep-learning models (SOTA) trained individually on each dataset for each task as reference benchmarks. Since these models are tailor-made to specific tasks, they are used only for comparison not as baselines. Details of these models can be found in the Appendix B.2 . 3 Experiments In this section, we have reported the details of various experiments conducted as a part of this study. Particularly, we conducted three types of experiments, each based on a unique research philosophy. While the first experiment prompts the VLMs using various prompting strategies, the second experiment finetunes the LoRA adapter associated with the VLMs. Lastly, in the third experiment, we finetuned the LLMs by giving VLM-generated meme understanding as input. We reported the details of the individual experiment in subsections. 3.1 Experiment-1 (Exp1): Prompting VLMs using various methods In this experiment, we evaluated various prompting strategies such as ZS , ZSC , FS , and FSC to classify memes. While in the ZS setting we prompted VLMs to classify the memes, in ZSC Wei et al. (2022) , Xu et al. (2024) we additionally asked the VLMs to provide detailed step-by-step reasoning. In the FS version, we followed similar steps; however, in addition, we included some example input for context. The FSC setting combined the FS approach with CoT-based reasoning. Prompting has been shown to influence model outputs across a range of visual reasoning tasks Zhou et al. (2024) . More details are provided in the Appendix B.3 . 3.2 Experiment-2: Fine-tuning VLMs using LoRA adapter In this experiment, we explored a substitute option of employing LoRA (Low-Rank Adaptation) Hu et al. (2022) , a parameter-efficient approach for fine-tuning VLMs. Instead of updating all model parameters, LoRA trains lightweight adapter layers, enabling VLMs to efficiently adapt to the meme classification task. Further details of the LoRA method are provided in the Appendix B.4 . Since the performance of the pre-trained VLMs depends on the training data, they perform well only if the model is trained on similar data. 3.3 Experiment-3: Combining VLM Explanation to Fine-tune LLMs In this experiment, CoVExFiL, we incorporated a novel two-step approach where, i) first, we generated textual understanding of the memes by prompting VLMs, and subsequently, ii) we fine-tuned the LLMs (details in Appendix B.2.1 ) using the scene understanding generated by the VLMs. Our objective was to investigate whether VLM-generated understanding explanations can help improve the LLMs to classify the memes. In

Read the complete text