Beyond Meme Templates: Limitations of Visual Similarity Measures in Meme Matching
Muzhaffar Hazman, Susan McKeever, Josephine Griffith · 2025
Read Beyond Meme Templates: Limitations of Visual Similarity Measures in Meme Matching by Muzhaffar Hazman, Susan McKeever, Josephine Griffith (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03562.txt.
Introduction
Muzhaffar Hazman, Susan McKeever, and Josephine Griffith test how well image similarity tools match memes that share no template. Most meme matching assumes a shared background picture with shifting text, so tools compare backgrounds alone. Many memes break that mold, and dictionary linking needs the wider view. The team frames a broader matching task and trials standard similarity measures on it.
Conventional measures excelled on template memes and fell short on other formats. A segment-wise version of the measures beat whole-image scoring on non-template memes every time. A prompting method with a multimodal language model is explored as a further route.
Catalog record
| CREATOR | Muzhaffar Hazman, Susan McKeever, Josephine Griffith |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Meme Templates: Limitations of Visual Similarity Measures in Meme Matching Thanks: Muzhaffar Hazman, Susan McKeever, Josephine Griffith Abstract Internet memes, now a staple of digital communication, play a pivotal role in how users engage within online communities and allow researchers to gain insight into contemporary digital culture. These engaging user-generated content are characterised by their reuse of visual elements also found in other memes. Matching instances of memes via these shared visual elements, called meme matching , is the basis of a wealth of meme analysis approaches. However, most existing methods assume that every meme consists of a shared visual background, called a template , with some overlaid text, thereby limiting meme matching to comparing the background image alone. Current approaches exclude the many memes that are not template-based and limit the effectiveness of automated meme analysis and would not be effective at linking memes to contemporary web-based meme dictionaries. In this work, we introduce a broader formulation of meme matching that extends beyond template matching. We show that conventional similarity measures, including a novel segment‑wise computation of the similarity measures, excel at matching template-based memes but fall short when applied to non-template-based meme formats. However, the segment-wise approach was found to consistently outperform the whole‑image measures on matching non-template-based memes. Finally, we explore a prompting-based approach using a pretrained Multimodal Large Language Model for meme matching. Our results highlight that accurately matching memes via shared visual elements, not just background templates, remains an open challenge that requires more sophisticated matching techniques. I Introduction Multimodal Internet memes (more commonly, memes) have become a staple medium of communication in almost all online digital communities. Like various forms of multimodal user-generated content, memes combine visual and textual elements to create creative and visually engaging expressions. Uniquely, memes are distinguished from other forms of digital media by their use of shared visual elements also found in other memes [ 1 ] , forming groups of related memes that express meaning using these elements. We refer to the task of identifying pairs or groups of memes as being related as meme matching approaches. Meme matching provides context for analysing individual memes [ 2 , 3 ] and enables the analysis of memes as networks of interconnected expressions [ 4 , 5 ] . Many current meme matching approaches are limited to memes that use shared visual backgrounds ( meme templates ), where two memes are deemed related only if they share the same visual background while other elements are ignored; we refer to this as Template Matching ( TM ). These approaches require (1) a visual representation, (2) a quantitative similarity measure, and (3) a threshold that determines whether a meme pair qualifies as a match. Commonly used similarity measures include the distance between image hashes [ 4 ] , cosine similarity between embeddings from a pretrained visual encoder [ 6 ] , and the distance between descriptors of visual keypoints [ 5 ] . However, the relationship between memes is not only defined based on common templates [ 1 ] . Hazman et al. defined a meme as “a user-generated piece of digital content that comprises either visual and/or textual components that can be identified as memetic elements” [ 2 ] . Building on this, we define Memetic Matching ( MM ) as the task of determining whether two memes share a visual memetic element. Memetic elements [ 2 ] include characters, standalone images and superimposed foreground elements [ 7 , 8 ] , as well as background templates. Although various visual similarity measures have been evaluated on TM [ 3 ] , we found no similar analysis for the MM . Currently, studies on memes, e.g. [ 4 , 6 , 9 ] , tend to rely on a Template Matching step. Since memetic elements take many forms, these analyses ignore a significant portion of the meme typology. Recently, Hazman et al. found that only 55% of memes randomly sampled from various meme classification datasets were template-based [ 2 ] . Similarly, knowledge-enriched meme analyses that use TM to retrieve semantic information can only access a subset of any knowledge base which reflect this diversity, such as community-maintained meme dictionaries [ 9 ] and knowledge graphs [ 10 ] . Accurately matching memes with relevant knowledge base entries on various memetic elements requires meme matching methods that go beyond TM . Image A Image B Memetic Matching ( MM ) Template Matching ( TM ) 1 related unrelated 2 related related 3 related unrelated 4 related unrelated TABLE I: Sample Image Pairs with class labels under the MM and TM tasks. In this work, we propose Memetic Matching as a more comprehensive formulation of meme matching than Template Matching. We compare the performance of various visual similarity measures in tackling the two tasks. Each task is a binary classification problem to determine if a meme is related to (matches) a given reference image, i.e., a dictionary entry image for MM and a template image for TM . The similarity measures are computed using visual representations that are commonly used in existing TM approaches: visual keypoints, pretrained embeddings, and image hashes. We also propose and evaluate a novel segment‐wise representation of embedding and hash similarities. We also evaluate the performance of a pretrained Multimodal LLM (MLLM) in tackling the Memetic Matching task. To evaluate these methods on tackling MM and TM , we prepare a manually labelled dataset for each task. Our findings reveal several key insights. First, MM is shown to be distinct from and more challenging than TM ; when tackled using the visual similarity measures. Second,the two tasks are best approached using different similarity measures. Furthermore, we found that segment-wise similarity measures outperformed their whole-image counterparts for MM . Finally, we show that zero-shot and few-shot prompting of an MLLM yield even poorer results for MM . These findings underscore the greater complexity of Memetic Matching and point to the need for more advanced solutions to reliably link memes to entries in community-maintained meme dictionaries. II Datasets For the two tasks, Template Matching and Memetic Matching, we create separate datasets, \mathcal{D}^{\texttt{TM}} and \mathcal{D}^{\texttt{MM}} , respectively. Each row of these datasets is a tuple: (m,r,\hat{y})\in\mathcal{D}^{(t)} where m is a meme image, r is either a dictionary entry image (for t=\texttt{MM} ) or a template image (for t=\texttt{TM} ), and \hat{y} indicates whether the pair is related or unrelated per the definition used for the task t . II-A Task 1: Template Matching ( TM ) As is common amongst existing TM works, e.g. [ 3 ] , a template-based definition for meme matching only considers whether a meme m uses a given template r^{\texttt{TM}} as its background. To prepare \mathcal{D}^{\texttt{TM}} , we replicate the data source, collection, and annotation methods reported in [ 5 , 3 ] . We collected 2,016 templates listed on IMGFlip.com. Similar to [ 3 ] , we removed duplicate templates by extracting all templates with a ResNext embedding cosine similarity greater than 85% to another template. These were manually reviewed for groups of duplicates, and all but one duplicate in each group was removed. Cropped versions of templates were also considered duplicates. Our final set