Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques

Lang Xiong, Raina Gao, Alyssa Jeong, Yicheng Fu, Vasu Sharma, Kevin Zhu · 2025

Read Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques by Lang Xiong, Raina Gao, Alyssa Jeong, Yicheng Fu, Vasu Sharma, Kevin Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00658.txt.

Introduction

Raina Gao and co-authors introduce, in a 2025 paper, a benchmark for how well language models handle sarcasm. Sarcasm says the opposite of what it means, which makes it hard for machines to read and to produce. Their benchmark sorts sarcasm into seven types, from deadpan and polite to raging and manic.

They labeled entries from an existing sarcasm data set by type. Then they tested models under several prompting styles, from no examples to step by step reasoning. They also propose a new prompt style built on emotion, and a generation method drawn from sarcasm's key parts: a clash of meaning, shock value, and context. The best classifier used the emotion prompts, though its score stayed modest. Human judges liked the emotion based generations much more than plain ones.

Catalog record

CREATORLang Xiong, Raina Gao, Alyssa Jeong, Yicheng Fu, Vasu Sharma, Kevin Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00658

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques Lang Xiong, Raina Gao, Alyssa Jeong, Yicheng Fu, Vasu Sharma, Kevin Zhu Abstract Sarcasm is a form of humor where expressions convey meanings opposite to their literal interpretations. Classifying and generating sarcasm using large language models is vital for interpreting human communication. Sarcasm poses challenges for computational models, due to its nuanced nature. We introduce Sarc7, a benchmark that classifies 7 types of sarcasm: self-deprecating, brooding, deadpan, polite, obnoxious, raging, and manic by annotating entries of the MUStARD dataset. Classification was evaluated using zero-shot, few-shot, chain-of-thought (CoT), and a novel emotion-based prompting technique. We propose an emotion-based generation method developed by identifying key components of sarcasm–incongruity, shock value, and context dependency. Our classification experiments show that Gemini 2.5, using emotion-based prompting, outperforms other setups with an F1 score of 0.3664. Human evaluators preferred our emotion-based prompting, with 38.46% more successful generations than zero-shot prompting. 1 Introduction Sarcasm is defined as the use of remarks that convey the opposite of their literal meaning. Understanding sarcasm requires an intuitive grasp of humor and social cues, which are essential for advanced natural language processing (NLP) tasks such as human-like conversation Yao et al. 2024 ; Gole et al. 2024 . In general, LLMs perform poorly in both sarcasm classification and generation, largely due to its subtlety and context dependence Yao et al. 2024 . Traditional sentiment analysis and machine learning techniques also struggle with these challenges. Sarcasm generation—which involves producing contextually appropriate and coherent sarcastic responses—remains particularly difficult Zhang et al. 2024 . Existing models often fail due to sarcasm’s reliance on social complexity and contradiction Gole et al. 2024 . Early generation approaches were rule-based and template-driven, resulting in rigid outputs Zhang et al. 2024 . Deep learning models trained on sarcastic corporate improved adaptability but still struggle with subtlety and contextual relevance Zhang et al. 2024 . This work introduces a novel sarcasm benchmark based on the seven types of sarcasm and introduces an emotion-based technique for both classification and generation. 2 Related Work Previously, SarcasmBench Zhang et al. 2024 established benchmarks for binary sarcasm classification by evaluating SOTA LLMs and PLMs. Current benchmarks do not address specific sarcasm type classification or generation, or emotion as a controlled factor. Emotion and sarcasm are directly correlated, as sarcasm is emotionally fueled and reflects the speaker’s emotion, both intentionally and unintentionally Leggitt and Gibbs 2000 ; Biswas et al. 2019 . Linguists have identified seven basic types of sarcasm, which we use to build our Sarc7 benchmark Qasim 2021 . Sarcasm Classification: Recent advances have focused on structured prompting techniques that use pragmatic reasoning to enhance sarcasm detection Lee et al. 2024 . Approaches such as pragmatic metacognitive prompting method (PMP) have improved model performance by making sarcasm inference more explicit Yao et al. 2024 ; Lee et al. 2024 . Furthermore, recent studies have shown that integrating commonsense, knowledge, and attention mechanisms help models identify subtleties in sarcastic statements Zhuang et al. 2025 . These methods show that guiding LLMs with structured signals can help them better understand the nuances of sarcastic statements. Sarcasm Generation: Recent studies have introduced controlled generation methods to guide LLMs toward producing sarcastic statements using contradiction strategies and dialogue cues Zhang et al. 2024 ; Helal et al. 2024 . Structured prompting and contradiction-based strategies have shown to improve sarcasm generation. Some methods guide LLMs by introducing contrast between expected and actual meanings or using contextual dialogue cues for coherence Zhang et al. 2024 ; Helal et al. 2024 ; Skalicky and Crossley 2018 . However, existing techniques struggle with controlling sarcasm levels and aligning them with contextual incongruence, shock value, and prior context dependency. 3 Methods 3.1 Benchmark Construction We introduce Sarc7 , a novel benchmark for fine-grained sarcasm classification and generation. Building on the MUStARD dataset ( Castro et al. 2019 ) , which provides binary sarcasm annotations for short dialogue segments, we manually annotated each sarcastic utterance with one of seven distinct sarcasm types: self-deprecating , brooding , deadpan , polite , obnoxious , raging , and manic . Qasim 2021 . These labels capture the variation in tone, intent, and emotional affect beyond binary sarcasm detection. Figure 1 shows the distribution of the seven annotated sarcasm types. All annotations were performed by humans using detailed definitions and examples to ensure consistency. The annotation process is illustrated in Figure 2 . Figure 1: Distribution of annotation labels in the dataset. The resulting Sarc7 benchmark supports two tasks: (1) multi-class sarcasm classification, and (2) sarcasm-type-conditioned generation. These tasks allow for more fine-grained evaluation of sarcasm understanding in large language models. Figure 2: Flowchart of the Step-by-Step Process for Sarcasm Classification Annotation 3.2 Task Definition We define two primary evaluation tasks: • Sarcasm Classification : Given a sarcastic utterance and its dialogue context, correctly predict the dominant sarcasm type from among the seven annotated categories. • Sarcasm Generation : Generate a sarcastic utterance consistent with one of the 7 types of sarcasm. Table 1 outlines definitions for each sarcasm category in the Sarc7 benchmark. Type Definition Self-deprecating Critique of oneself. Brooding Frustration masked as politeness. Deadpan Flat delivery with hidden meaning. Polite Insincere statements delivered as courtesy. Obnoxious Deliberately rude or provocative. Raging Extreme exaggeration to express anger. Manic Overexcited or erratic expression. Table 1: Definitions of the seven sarcasm types 3.3 Baseline Classification Our baseline testing focused on zero-shot, few-shot, and CoT prompting. For generations, baseline outputs were produced using a zero-shot prompt, without structured control over dimensions. These baselines were evaluated by a human grader based on accuracy of sarcasm type and emotion. 3.4 Emotion-Based Prompting Emotion-based prompting refers to the assistance of emotion categories per sarcasm type. American psychologist Paul Ekman identified six basic emotions: happiness, sadness, anger, fear, disgust, and surprise Ekman 1992 . Our emotion-based prompting technique consists of two main steps: 1) Categorize the emotion of the context. 2) Classify the emotion of the utterance. 3.5 Generation Dimensions Our approach moves beyond general sarcasm generation by focusing on four dimensions to generate specific types of sarcasm: Each generation was assigned three independent scores and one category: • Incongruity : Degree of semantic mismatch. • Shock Value : Intensity of sarcasm. • Context Dependency : Reliance on conversational history. • Emotion : one of the six basic emotions By incorporating these dimensions, our work improves sarcasm generation beyond existing contradiction-based approaches, allowing for more precise, context-aware, and nuanced sarcasm that satisfies one of the seven types of sarcasm. A sample of a generation using the emotion-based technique is in Figure 3 . Figure 3: Sample output using emotion-based method 4 Experiments 4.1 Model Selection We evaluate several state-of-the-art language models on our proposed sarcasm benchmark, including GPT-4o OpenAI 2024 , Claude 3.5 Sonnet Anthropic 2024 , Gemini 2.5 DeepMind et al. 2023 , Qwen 2.5 Team 2024 , and Llama 4 Maverick. 4.2 Evaluation We evaluated classification by comparing model predictions to human-annotated labels across seven sarcasm types. For generation, GPT-4o produced 100 sarcastic statements per prompting method, each rated by a human for type accuracy. 5 Results Model 0-shot Few-shot CoT Emotion-based GPT-4o 47.73% 50.29% 55.07% 48.94% Claude

Read the complete text