PC-MNet: Dual-Level Congruity Modeling for Multimodal Sarcasm Detection via Polarity-Modulated Attention Maoheng Li, Ling Zhou, Xiaohua Huang, Rubing Huang, Wenming Zheng, Guoying Zhao Abstract Multimodal sarcasm detection, which aims to precisely identify pragmatic incongruities between literal text and nonverbal cues, has gained substantial attention in multimodal understanding. Recent advancements have predominantly relied on naïve similarity-based attention mechanisms and uniform late fusion strategies. However, sarcastic utterances inherently exhibit polar-opposite semantic correlations rather than similarity. Consequently, employing similarity-seeking mechanisms inevitably fails to capture cross-modal contradictions, resulting in environmental noise, severe feature redundancy, and functional entanglement. To address these critical limitations, we propose the Polarity-Congruity Multimodal Network (PC-MNet), a novel hierarchical multi-granularity verification framework. Specifically, we first introduce a region-guided semantic alignment pipeline and a polarity-modulated attention mechanism to eradicate irrelevant visual noise and mathematically enforce the amplification of atomic-level cross-modal contradictions. This decoupling enables the extraction of the sracasm feature via parallel bipartite-dominant heterogeneous graphs. Furthermore, given that functional entanglement restricts traditional late fusions, we incorporate a scalar congruity routing mechanism and a prior-guided contextual graph. This mechanism anchors a generalized incongruity manifold through a two-stage asymmetric optimization driven by inconsistency-aware contrastive learning, selectively fusing only the most discriminative multi-granularity evidence. Extensive experiments on the MUStARD benchmark and its spurious-correlation-mitigated balanced datasets demonstrate that our approach achieves new state-of-the-art performance, surpassing the strongest multimodal baseline by a substantial 3.14% improvement in Macro-F1. By architecturally isolating atomic, composition, and contextual conflicts. This work provides a robust, decoupled paradigm for modeling subtle pragmatic incongruities in human communication. I Introduction Multimodal sentiment analysis plays a critical role in human-computer interaction and opinion mining. However, multimodal sarcasm detection remains highly challenging, due to the subtlety and ambiguity of pragmatic reasoning across different modalities. Sarcasm is a pragmatic phenomenon in which speakers convey sentiments contrary to their literal words [ 2 ] [ 6 ] . As shown in Figure 1 , prior studies [ 3 , 7 ] identify two mechanisms of sarcasm: modal incongruity and contextual incongruity . Modal incongruity involves explicit cross-modal contradictions within an utterance, such as positive words accompanied by a dismissive tone [ 21 ] . Conversely, contextual incongruity occurs when an internally consistent utterance contradicts the conversational context or speaker characteristics [ 11 ] . Fig. 1 : Modal and contextual incongruity in multimodal sarcasm. The top example illustrates modal incongruity through conflicting polarities among text, visual, and acoustic signals within a single utterance. The bottom example demonstrates contextual incongruity, where a multimodally neutral utterance contradicts the preceding conversational context. To address these challenges, current methods typically rely on feature fusion and interaction. Several studies employ tensor-based fusion [ 36 ] or cross-modal attention [ 21 ] to align multimodal sequences with text. Meanwhile, other approaches utilize Graph Neural Networks (GNNs) [ 10 ] to capture global conversational structures. However, effectively modeling both forms of incongruity remains difficult due to two critical limitations. First, traditional cross-modal attention focuses on modality similarity rather than contradiction , often failing to capture the negative correlations that indicate sarcasm [ 21 ] . Second, existing graph-based methods often entangle local intra-utterance conflicts with global inter-utterance dynamics. Consequently, this structural entanglement and the inherent lack of adaptability in dynamically shifting focus between modal and contextual incongruities severely limit the generalization capabilities of such models. To alleviate these limitations, we propose the Polarity-Congruity Multimodal Network (PC-MNet), a hierarchical framework that explicitly captures contradictory multimodal cues. Unlike similarity-based methods, PC-MNet employs a decoupled reasoning pipeline to avoid feature homogenization. First, a region-guided semantic alignment module isolates human subjects to mitigate environmental noise and projects the refined visual cues into a cross-modally aligned semantic space to facilitate contradiction detection. Then, a polarity-modulated attention mechanism injects learnable polarity distances to capture fine-grained cross-modal contradictions. Furthermore, we construct parallel heterogeneous graphs to extract intra-utterance sarcasm features, offering an alternative to standard Transformer-based modeling. We also introduce a scalar congruity routing mechanism to isolate high-dimensional topological information, yielding scalar sarcasm priors. These priors then guide a downstream contextual graph to model conversational shifts. Finally, an adaptive dual-granularity fusion layer dynamically aggregates micro-level and macro-level evidence. The whole model is optimized via a two-stage strategy, transitioning from a valence-guided initialization to an inconsistency-aware contrastive loss. In summary, our main contributions are as follows: • We propose PC-MNet, a hierarchical framework that addresses the feature entanglement problem in existing multimodal sarcasm detection models. By decoupling the reasoning pipeline, it effectively captures both micro-level modal and macro-level contextual incongruities. • We propose a polarity-modulated attention mechanism and a prior-guided contextual graph neural network. These modules adaptively model contradiction-driven topologies, capturing both fine-grained cross-modal and inter-utterance contradictions. • We design a scalar congruity routing mechanism to extract structural sarcasm priors. Additionally, we employ a two-stage optimization scheme with an inconsistency-aware contrastive loss to learn robust representations and prevent feature collapse. • Experimental results on the MUStARD benchmark and the framework balanced datasets validate the effectiveness of PC-MNet. It outperforms existing fusion methods and large multimodal models, achieving a state-of-the-art Macro-averaged F1-score of 81.64%, as supported by comprehensive ablation studies. II Related Work This section reviews the evolution of computational sarcasm detection from text-based methods to multimodal paradigms, highlighting the structural limitations that our approach aims at addressing. II-A Multimodal Sentiment Analysis and Sarcasm Detection Early text-based sarcasm detection evolved from rule-based lexicons [ 23 ] to deep sequential models [ 25 ] and bidirectional transformers like BERT [ 4 ] . However, text-only methods often fail when literal meanings are contradicted by nonverbal cues. Consequently, multimodal sentiment analysis emerged, utilizing early outer-product fusions [ 36 ] or cross-attention and multi-view architectures [ 21 , 31 , 27 ] to optimize modality consistency. However, sarcasm inherently arises from affective incongruity [ 2 ] . Regarding incongruity, extracting visual features directly from raw frames [ 1 ] or computing CLIP similarity can introduce environmental noise and semantic alignment bias. To address modality conflict, methods such as D&R Net [ 30 ] and Att-BERT [ 1 ] employ discrepancy-aware attention. Nevertheless, relying on late-fusion strategies that rigidly concatenate intermediate representations often leads to feature redundancy. Furthermore, Multimodal Transformers that focus on similarity tend to homogenize features, obscuring essential cross-modal boundaries. Graph Neural Networks (GNNs) offer topological alternatives that preserve modality identities [ 29 , 16 , 28 ] . Yet they often entangle modalities into monolithic graphs, leading to semantic over-smoothing and routing messages solely based on native feature similarity. Additionally, integrating external knowledge bases such as ConceptNet [ 35 ] limits flexibility due to their static rules. PC-MNet is designed to address these structural limitations. We use object detection to isolate human expressive features, thereby mitigating background noise. Instead of optimizing for alignment similarity, our polarity-modulated attention explicitly models cross-modal contradictions. To prevent similarity-based homogenization, we extract intra-utterance sarcasm features using parallel bipartite-dominant heterogeneous graphs, in which edges are weighted by polarity distances rather than simple feature similarity. Departing from late fusion, an adaptive dual-granularity fusion utilizes a scalar congruity routing mechanism to filter high-dimensional representations, using them exclusively as structural priors or contrastive optimization anchors. Finally, a two-stage asymmetric optimization uses continuous valence scores to warm up the model before applying contrastive supervision. II-B Vision-Language Models and Fine-Grained Verification Recent benchmarks evaluate zero-shot and few-shot reasoning in large vision-language models using prompt-based paradigms for pragmatic deduction [ 38 , 15 , 37 , 34 ] . Despite their scale, the autoregressive generation and flattened-sequence modeling inherent in these models limit their ability to perform explicit, fine-grained cross-modal verification. Without structural constraints to isolate micro-level polarity conflicts, these models are prone to hallucinations and precision-recall imbalances, often overpredicting sarcasm based solely on the global semantic context. PC-MNet addresses this architectural deficiency by introducing topological priors. By structurally decoupling micro-level conflicts, macro-level sarcasm features, and conversational dynamics, our network explicitly verifies fine-grained pragmatic incongruities rather than relying entirely on parameter scaling. Fig. 2 : The overall architectural framework of PC-MNet. III Methodology The architecture of our Polarity-Congruity Multimodal Network is depicted in Figure 2 , and the overall forward and optimization pipeline is formally summarized in Algorithm 1 . Details of each constituent module are provided in the following sections. III-A Problem Formulation Formally, the framework receives a target utterance \mathbf{U}_{m} across aligned modalities m\in\{T,A,V\} . Alongside the target utterance, the network processes the corresponding conversational history, denoted as \mathbf{C}_{m}=[\mathbf{C}_{m,1},\dots,\mathbf{C}_{m,J}] , where J represents the fixed context length. The primary objective is to predict the binary sarcasm label: \small\hat{y}=P(y=1\mid\{\mathbf{U}_{m},\mathbf{C}_{m}\}_{m\in\{T,A,V\}}), (1) where y\in\{0,1\} indicates a non-sarcastic or sarcastic state, respectively. Unlike conventional methods that blindly extract features, our approach utilizes explicit, continuous valence scores v\in[-1,1] . Advanced multimodal benchmark annotations natively provide these valence scores. We utilize the native scores to enable a rigorous, context-aware cold-start for the polarity manifold. III-B Step 1: Multimodal Feature Encoding We use BERT and Wav2Vec 2.0-base to extract sequential features \mathbf{U}_{T}\in\mathbb{R}^{L_{T}\times d_{T}} and \mathbf{U}_{A}\in\mathbb{R}^{L_{A}\times d_{A}} . The variables L_{T} and L_{A} denote the sequence lengths of the textual tokens and acoustic frames, while d_{T} and d_{A} represent their respective original feature dimensions. To prevent background clutter from diluting pragmatic signals, a region-guided semantic alignment pipeline extracts focal subjects via the following two stages. Stage 1 uses YOLOv8 to detect human bounding boxes, thereby eliminating non-human noise and triggering deterministic zero-padding for human-absent frames. Stage 2 independently encodes retained crops via CLIP ViT-B/32, strictly prohibiting text-guided semantic filtering to preserve unadulterated crossmodal contradictions. The implemented detection-driven strategy effectively captures active speakers. Consequently, the pipeline excludes environmental noise while ensuring extracted visual embeddings \mathbf{U}_{V}\in\mathbb{R}^{L_{V}\times K\times d_{V}} and context nodes \mathbf{C}_{V,j}\in\mathbb{R}^{d_{V}} , padded to maximum K subjects per frame, remain authentic. Applying a position-wise Multi-Layer Perceptron (MLP), we flatten visual spatio-temporal axes to L_{V}^{eff}=L_{V}\times K , mapping 2D sequences into a shared semantic space: \small\mathbf{H}_{m}=\text{Dropout}\left(\text{ReLU}\left(\text{LayerNorm}(\mathbf{U}_{m}\mathbf{W}_{m}+\mathbf{b}_{m})\right)\right), (2) where \mathbf{H}_{m}\in\mathbb{R}^{L_{m}^{eff}\times d_{enc}} represents the aligned sequence. For the historical context, we retain native sequences \mathbf{H}_{m,j} for subsequent attention mechanisms while extracting utterance-level anchors via temporal mean-pooling: \mathbf{h}_{m,j}^{c}=\text{MeanPool}(\mathbf{H}_{m,j})\in\mathbb{R}^{d_{enc}} . Input: Batch \mathcal{B}=\{(\mathbf{U}_{m,i},\mathbf{C}_{m,i},y_{i})\}_{i=1}^{|\mathcal{B}|} for m\in\{T,A,V\} , Stage parameter Output: Predictions \mathcal{Y} , Total Loss \mathcal{L}_{total} \mathcal{Y}\leftarrow\emptyset ; for each instance i\in\mathcal{B} do Encode native sequences \{\mathbf{H}_{m,i}\} , history \{\mathbf{H}_{m,j,i}\} , and anchors \{\mathbf{h}_{m,j,i}^{c}\}_{j=1}^{J} ; (\mathbf{e}_{atomic,i},\mathbf{s}_{comp,i},\mathbf{z}_{incon}^{(i)})\leftarrow Atomic & Composition Congruity described in Alg. 2 ; Extract target base state \mathbf{h}_{tgt,i}^{(0)} via temporal pooling; \mathbf{e}_{inter,i}\leftarrow Prior-Guided Contextual RGAT described in Alg. 3 ; \mathbf{M}_{fuse}\leftarrow[\text{Proj}(\mathbf{e}_{atomic,i});\text{Proj}(\mathbf{e}_{inter,i})] ; \mathbf{a}_{fuse}\leftarrow\text{Softmax}(\mathbf{w}_{fuse}^{\top}\tanh(\mathbf{M}_{fuse}\mathbf{W}_{F})) ; \hat{y}_{i}\leftarrow\text{Classifier}(\mathbf{M}_{fuse}^{\top}\mathbf{a}_{fuse}) ; \mathcal{Y}\leftarrow\mathcal{Y}\cup\{\hat{y}_{i}\} ; Compute \mathcal{L}_{cls} using \mathcal{Y} and ground truth \{y_{i}\} ; Compute \mathcal{L}_{con} using manifold representations \{\mathbf{z}_{incon}^{(i)}\} ; if Stage == 1 (Cold-Start) then Compute \mathcal{L}_{valence} using native valence scores; \mathcal{L}_{total}\leftarrow\lambda_{cls}\mathcal{L}_{cls}+\lambda_{con}\mathcal{L}_{con}+\lambda_{val}\mathcal{L}_{valence} ; else \mathcal{L}_{total}\leftarrow\lambda_{cls}\mathcal{L}_{cls}+\lambda_{con}\mathcal{L}_{con} return \mathcal{Y},\mathcal{L}_{total} ; Algorithm 1 PC-MNet Forward & Optimization Pipeline III-C Step 2: Atomic-Level Congruity The detailed in Algorithm 2 , explicitly modeling cross-modal contradictions begins by designating text as query q=T and non-verbal modalities as keys k\in\{A,V\} , projecting sequences into a shared polarity space: \displaystyle\mathbf{P}_{T} \displaystyle=\text{L2Norm}(\text{MLP}_{pol}^{atomic}(\mathbf{H}_{T})) (3) \displaystyle\mathbf{P}_{k} \displaystyle=\text{L2Norm}(\text{MLP}_{pol}^{atomic}(\mathbf{H}_{k})). Sharing the weights of the \text{MLP}_{pol} module across modalities ensures consistent angular measurements. During the initial cold-start phase, these projected polarities are directly supervised by continuous valence scores (as detailed in Section III-F ). Consequently, this explicit supervision guarantees that the formulated contradiction matrix \mathbf{C}^{T\to k}=\mathbf{1}-\mathbf{P}_{T}\mathbf{P}_{k}^{\top} accurately quantifies the pragmatic divergence between verbal and non-verbal signals. Higher scores structurally modulate the attention score \mathbf{S} : \small\mathbf{S}=\text{MHA\_Scores}(\mathbf{H}_{T},\mathbf{H}_{k})+\alpha_{mic}\mathbf{C}^{T\to k}. (4) Learnable amplifier \alpha_{mic} controls micro-conflict sensitivity. Applying topological mask \mathbf{M}_{mask} before Softmax and incorporating a residual connection yields the atomic-level conflict representation \mathbf{E}_{atomic}^{T\to k}=\text{Softmax}(\mathbf{S}+\mathbf{M}_{mask})\mathbf{H}_{k}+\mathbf{H}_{T} , which routes attention to contradictory pairs while preserving intrinsic textual semantics. To extract the global micro-conflict signal, we apply temporal mean-pooling to both non-verbal branches and concatenate them, yielding the final unified atomic vector \mathbf{e}_{atomic} : \small\mathbf{e}_{atomic}=[\text{MeanPool}(\mathbf{E}_{atomic}^{T\to A})\parallel\text{MeanPool}.(\mathbf{E}_{atomic}^{T\to V})]. (5) Input: Target features \mathbf{H}_{T},\mathbf{H}_{A},\mathbf{H}_{V} Output: Atomic vector \mathbf{e}_{atomic} , Tension prior \mathbf{s}_{comp} , Incongruity rep. \mathbf{z}_{incon} Project sequences to shared polarity space \mathbf{P}_{m},\forall m\in\{T,A,V\} ; for k\in\{A,V\} do Calculate contradiction matrix \mathbf{C}^{T\to k}\leftarrow\mathbf{1}-\mathbf{P}_{T}\mathbf{P}_{k}^{\top} ; \mathbf{E}_{atomic}^{T\to k}\leftarrow\text{PolarityModulatedMHA}(\mathbf{H}_{T},\mathbf{H}_{k},\mathbf{C}^{T\to k},\alpha_{mic}) ; \mathbf{e}_{atomic}\leftarrow\text{MeanPool}(\mathbf{E}_{atomic}^{T\to A})\parallel\text{MeanPool}(\mathbf{E}_{atomic}^{T\to V}) ; for target pair (q,k)\in\{(T,A),(T,V)\} do Initialize joint node matrix \mathbf{H}_{joint}^{(0)} and polarity-modulated adjacency \mathbf{A}_{mod} ; \mathbf{H}_{joint}^{(L_{mac})}\leftarrow\text{L-layer GCN}(\mathbf{H}_{joint}^{(0)},\mathbf{A}_{mod}) ; Extract composition vectors \mathbf{h}_{q}^{comp},\mathbf{h}_{k}^{comp} from \mathbf{H}_{joint}^{(L_{mac})} ; Compute scalar tension s_{comp}^{qk} and discrepancy vector \boldsymbol{\Delta}_{qk} ; \mathbf{s}_{comp}\leftarrow[s_{comp}^{TA},s_{comp}^{TV}] ; \mathbf{z}_{incon}\leftarrow\text{L2Norm}(\text{MLP}_{diff}([\boldsymbol{\Delta}_{TA}\parallel\boldsymbol{\Delta}_{TV}])) ; return \mathbf{e}_{atomic},\mathbf{s}_{comp},\mathbf{z}_{incon} ; Algorithm 2 Atomic & Composition Congruity III-D Step 3: Composition-Level Congruity To prevent the entanglement and over-smoothing inherent in dense Transformers, constructing parallel, bipartite-dominant heterogeneous graphs preserves identities and redefines message passing in response to polarity contradictions. Building these graphs exclusively for the target utterance focuses reasoning, prevents computational explosion, and defers historical context to Step 4 (Section III-E ), enforcing a scalar congruity routing. Injecting learnable modality-specific type embeddings \mathbf{E}_{type}^{q} and \mathbf{E}_{type}^{k} maintains identities: \small\mathbf{H}_{joint}^{(0)}=[\mathbf{H}_{q}+\mathbf{E}_{type}^{q};\mathbf{H}_{k}+\mathbf{E}_{type}^{k}]. (6) Reusing the atomic polarity space ( \text{MLP}_{pol}^{atomic} ) established in Section III-C , projecting nodes into the shared manifold formulates the padding-masked adjacency matrix \mathbf{A}_{mod} via a bounded amplifier \alpha_{mac} . The mechanism structurally forces the GCN to route messages across highly contradictory cross-modal nodes, actively amplifying heterophilous pragmatic conflicts without smoothing intrinsic spatial-temporal topologies. Assigning zero weights for padded interactions isolates invalid nodes, preventing semantic pollution during graph convolution. After L_{mac} layers, partitioning and masked mean pooling yield macro representations \mathbf{h}_{q}^{comp} and \mathbf{h}_{k}^{comp} . Avoiding gradient interference from naive late fusion, a rigorous scalar congruity routing restricts composition-level features to two pathways. First, cosine congruity extracts purified macro-level features: \small s_{comp}^{qk}=\text{CosineSim}(\mathbf{h}_{q}^{comp},\mathbf{h}_{k}^{comp}). (7) Routing the O(1) prior s_{comp} averts O(d_{enc}) fusion explosions. Meanwhile, branch features bypass final fusion for contrastive learning exclusively, and local sliding windows preserve intra-modal continuity. III-E Step 4: Prior-Guided Contextual GNN Unlike standard conversational GNNs that blindly gather context, we introduce asymmetric node initialization and the injection of priors, as outlined in Algorithm 3 . To circumvent background noise in historical utterances j\in\{1,\dots,J\} , text-anchored alignment designates pooled text \mathbf{h}_{T,j}^{c} as a stable query extracting relevant cues from native acoustic ( \mathbf{H}_{A,j} ) and visual ( \mathbf{H}_{V,j} ) sequences via multi-head attention: \displaystyle\mathbf{H}_{AV,j} \displaystyle=[\mathbf{H}_{A,j}\parallel\mathbf{H}_{V,j}] (8) \displaystyle\mathbf{h}_{j}^{(0)} \displaystyle=\text{TextAnchored\_MHA}(\mathbf{Q}{=}\mathbf{h}_{T,j}^{c},\mathbf{K}{=}\mathbf{H}_{AV,j},\mathbf{V}{=}\mathbf{H}_{AV,j}). For the target utterance, distilling branch-specific features enforces the scalar congruity routing. We pool native sequences and project them to form the target base state \mathbf{h}_{tgt}^{(0)} . Injecting scalar prior \mathbf{s}_{comp} via LayerNorm and \mathbf{W}_{pri} creates the prior-aware target node \tilde{\mathbf{h}}_{tgt}^{(0)}=\mathbf{h}_{tgt}^{(0)}+\text{LayerNorm}(\mathbf{W}_{pri}\mathbf{s}_{comp}) . Concatenating this prior-aware target state with the historical nodes constructs the initial conversational feature matrix \mathbf{H}^{(0)}=[\mathbf{h}_{1}^{(0)};\dots;\mathbf{h}_{J}^{(0)};\tilde{\mathbf{h}}_{tgt}^{(0)}] . Three relations ( r_{seq} , r_{ctx} , r_{spk} ) yield binary masks \mathbf{A}^{r} . Projecting nodes into polarity space \mathbf{p}_{n} computes opposition distance C_{ij}=1-\mathbf{p}_{i}\cdot\mathbf{p}_{j} , forming relational score S_{ij,r}=\text{LeakyReLU}(\text{Linear}(\mathbf{H}_{i},\mathbf{H}_{j})+\alpha_{ctx}C_{ij}) . Normalizing over local neighborhood \mathcal{N}_{i}^{r} via masked softmax isolates valid pairs for relation weights \alpha_{ij,r}=\text{MaskedSoftmax}(S_{ij,r}\times\mathbf{A}_{i,j}^{r}) . Message passing updates features: \mathbf{H}_{i}^{(k)}=\text{ReLU}(\sum_{r}\sum_{j}\alpha_{ij,r}\mathbf{W}_{r}\mathbf{H}_{j}^{(k-1)}) . After K_{gnn} layers, the target node forms contextual embedding \mathbf{e}_{inter}=\mathbf{H}_{J+1}^{(K_{gnn})} . Input: History anchors and sequences \{\mathbf{h}_{T,j}^{c},\mathbf{H}_{A,j},\mathbf{H}_{V,j}\}_{j=1}^{J} , Target base \mathbf{h}_{tgt}^{(0)} , Prior \mathbf{s}_{comp} Output: Contextual representation \mathbf{e}_{inter} for j\leftarrow 1 to J do \mathbf{h}_{j}^{(0)}\leftarrow\text{TextAnchored\_MHA}(\mathbf{h}_{T,j}^{c},[\mathbf{H}_{A,j}\parallel\mathbf{H}_{V,j}]) ; Inject tension prior: \tilde{\mathbf{h}}_{tgt}^{(0)}\leftarrow\mathbf{h}_{tgt}^{(0)}+\text{LayerNorm}(\mathbf{W}_{pri}\mathbf{s}_{comp}) ; Initialize conversational topology \mathbf{H}^{(0)}\leftarrow[\mathbf{h}_{1}^{(0)};\dots;\mathbf{h}_{J}^{(0)};\tilde{\mathbf{h}}_{tgt}^{(0)}] ; for layer k\leftarrow 1 to K_{gnn} do Update node polarities \mathbf{p}_{n}\leftarrow\text{L2Norm}(\text{MLP}_{pol}^{ctx}(\mathbf{H}_{n}^{(k-1)})) ; for each relation type r\in\{seq,ctx,spk\} do Calculate contradiction penalties C_{ij}\leftarrow 1-\mathbf{p}_{i}\cdot\mathbf{p}_{j} ; Compute topology-masked attention \alpha_{ij,r} modulated by C_{ij} and \alpha_{ctx} ; \mathbf{H}_{i}^{(k)}\leftarrow\text{ReLU}\left(\sum_{j}\alpha_{ij,r}\mathbf{W}_{r}\mathbf{H}_{j}^{(k-1)}\right) ; return \mathbf{e}_{inter}\leftarrow\mathbf{H}_{J+1}^{(K_{gnn})} ; Algorithm 3 Prior-Guided Contextual RGAT III-F Step 5: Adaptive Fusion and Optimization To address heterogeneity in sarcasm, we fuse the atomic-level micro-conflict vector \mathbf{e}_{atomic} with the contextual representation \mathbf{e}_{inter} . Strictly excluding branch-specific graph features from fusion prevents semantic looping. Bridging scale discrepancies via a learnable scaling factor, independent GELU and LayerNorm projections form a dual-granularity matrix \mathbf{M}_{fuse}=[\text{Proj}(\mathbf{e}_{atomic});\text{Proj}(\mathbf{e}_{inter})] . A parameterized scorer dynamically routes the critical incongruity signal via attention weights \mathbf{a}_{fuse}=\text{Softmax}(\mathbf{w}_{fuse}^{\top}\tanh(\mathbf{M}_{fuse}\mathbf{W}_{F})) , yielding the final prediction \hat{y}=\text{Classifier}(\mathbf{M}_{fuse}^{\top}\mathbf{a}_{fuse}) . Standard cross-entropy \mathcal{L}_{cls} drives binary classification, while an inconsistency-aware supervised contrastive loss \mathcal{L}_{con} explicitly structures the representation space, forcing direct comparison of intrinsic structural conflicts. Projecting concatenated branch-specific cross-modal differences via an MLP yields an L2-normalized inconsistency representation \mathbf{z}_{incon}^{(i)} . For anchor i in batch \mathcal{B} , the contrastive loss clusters positive samples of the same class while pushing away all other valid samples: \small\small\mathcal{L}_{con}=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\frac{-1}{|\mathcal{P}(i)|}\sum_{p\in\mathcal{P}(i)}\log\frac{\exp(\mathbf{z}_{incon}^{(i)}\cdot\mathbf{z}_{incon}^{(p)}/\tau)}{\sum_{a\in\mathcal{A}(i)}\exp(\mathbf{z}_{incon}^{(i)}\cdot\mathbf{z}_{incon}^{(a)}/\tau)}, (9) where \mathcal{P}(i) is the positive set and \mathcal{A}(i) encompasses all valid samples excluding the anchor. To prevent mapping noise due to premature subclass separation, this contrastive objective aligns along a generalized incongruity manifold. Abstracting commonalities of sarcasm into a unified positive cluster anchors robust boundaries amid noisy signals, preventing semantic collapse. The two-stage asymmetric strategy optimizes the network. The initial cold-start phase anchors the polarity space by adding an MSE loss \mathcal{L}_{valence} computing the divergence between native continuous valence scores and a Tanh-bounded polarity probe output, preventing gradient explosion: \small\mathcal{L}_{total}=\lambda_{cls}\mathcal{L}_{cls}+\lambda_{con}\mathcal{L}_{con}+\lambda_{val}\mathcal{L}_{valence}. (10) Subsequently, detaching the auxiliary valence supervision transitions the network into topological refinement, relying solely on cross-entropy and contrastive losses: \small\mathcal{L}_{total}=\lambda_{cls}\mathcal{L}_{cls}+\lambda_{con}\mathcal{L}_{con}. (11) Weighting the contrastive loss heavily asserts that segregating conflict topologies remains fundamentally more critical than naive label-mapping gradients. IV Experimental Settings This section details the datasets, baseline models, and implementation details. IV-A Datasets We use MUStARD [ 3 ] as our foundational benchmark, a dataset specifically compiled for multimodal sarcasm detection, drawn from clips of popular television shows. To assess the model’s structural robustness and pragmatic reasoning capabilities, we extend the evaluation to two progressively challenging datasets. The first, MUStARD++ [ 22 ] , introduces fine-grained annotations and corrects surface labels. The second is MUStARD++ Balanced [ 37 ] . All datasets contain text, audio, and video modalities. Following standard affective computing protocols to prevent the memorization of specific facial and vocal features, we evaluate all models using a 5-fold cross-validation split. TABLE I : Comprehensive performance comparison on the MUStARD benchmark. Bold indicates best performance; underline indicates second-best. Category Model Prec (%) Rec (%) F1 (%) Acc (%) Text-Only SVM-TK [ 3 ] 69.40 71.20 70.30 72.60 BERT [ 4 ] 65.60 64.30 64.70 64.30 Tensor Fusion TFN [ 36 ] 66.10 64.20 65.30 66.50 LMF [ 5 ] 68.30 67.10 67.70 68.20 IKM [ 35 ] 70.10 69.30 69.70 70.50 Cross-Modal Attention MAG-BERT [ 21 ] 69.40 69.50 69.40 69.40 FiLM [ 9 ] 67.30 66.20 66.70 67.00 COSMOS [ 19 ] 70.80 69.10 69.90 70.20 Graph-Based ADG [ 18 ] 72.10 71.50 71.80 72.00 GCN-DN [ 10 ] 73.80 72.20 73.00 73.50 G 2 SAM [ 28 ] 75.30 73.60 73.50 73.60 GNN-CSO [ 24 ] 71.20 70.80 71.00 71.50 CMGCN [ 16 ] 74.92 72.25 71.58 72.37 Advanced Multitask ICON [ 12 ] 71.50 70.20 70.80 71.20 QPM [ 17 ] 77.50 77.60 77.50 77.50 MO-Sarcation [ 26 ] 77.90 77.90 77.90 77.90 CESDN [ 14 ] 76.20 74.20 75.20 75.50 VyAnG-Net [ 20 ] 78.80 78.20 78.50 79.90 ESAM [ 33 ] 71.95 68.97 69.05 71.05 Causal & Debiasing T-FCD [ 39 ] 73.50 72.80 73.10 73.40 CL-Debias [ 13 ] 74.10 73.50 73.80 74.00 MVIL [ 8 ] 75.80 74.90 75.30 75.60 Large Multimodal Models GPT-4o [ 32 ] 71.82 68.10 69.91 70.69 Llama 3-8B [ 32 ] 67.29 51.05 61.26 68.90 Qwen 2-7B [ 32 ] 70.98 54.35 61.46 65.94 Ours PC-MNet (V-Only) 63.40 62.10 62.74 63.50 PC-MNet (A-Only) 68.50 67.30 67.89 68.20 PC-MNet (T-Only) 75.40 74.80 75.09 75.80 PC-MNet (Full) 83.61 81.09 81.64 82.46 IV-B Baseline Models To rigorously evaluate the proposed PC-MNet, we comprehensively compare our approach against seven distinct categories of baselines, ranging from unimodal architectures to foundation models. The first category encompasses text-only models, specifically SVM-TK [ 3 ] and BERT [ 4 ] , which serve as foundational baselines that rely entirely on linguistic cues to detect sarcasm. Recognizing the necessity of non-verbal information, the second category includes tensor fusion approaches such as TFN [ 36 ] , LMF [ 5 ] , and IKM [ 35 ] . These models explicitly calculate outer products or integrate external knowledge to capture joint multimodal distributions. Subsequently, we evaluate cross-modal attention mechanisms, including MAG-BERT [ 21 ] , FiLM [ 9 ] , and COSMOS [ 19 ] , which leverage sophisticated alignment strategies and feature modulation to capture inter-modal dependencies and contextual grounding. Furthermore, given the topological nature of conversational dynamics, we benchmark against graph-based models, including ADG [ 18 ] , GCN-DN [ 10 ] , G 2 SAM [ 28 ] , CMGCN [ 16 ] , and GNN-CSO [ 24 ] . These architectures construct heterogeneous topological structures to explicitly model long-range contextual dependencies and cross-modal relational features. Additionally, advanced multitask frameworks including ICON [ 12 ] , QPM [ 17 ] , MO-Sarcation [ 26 ] , CESDN [ 14 ] , VyAnG-Net [ 20 ] , and ESAM [ 33 ] are included. These frameworks introduce auxiliary objectives, such as sentiment classification or sentiment constraints, to provide complementary supervisory signals for sarcasm detection. To address the critical issue of dataset artifacts, we also compare with causal debiasing methods, namely T-FCD [ 39 ] , CL-Debias [ 13 ] , and MVIL [ 8 ] , which mitigate spurious correlations through counterfactual reasoning or debiased contrastive learning. Notably, we explicitly reproduced the recent ESAM [ 33 ] model, yielding an overall accuracy of 71.05% and a Macro-F1 score of 69.05% under our experimental setting. IV-C Implementation Details In our experiments, we use BERT-large and Wav2Vec 2.0-base to extract textual and acoustic features, respectively. For the visual modality, we combine YOLOv8 with CLIP ViT-B/32. We set the shared encoding dimension to d_{enc}=512 and the polarity space dimension to d_{p}=16 . We configure the heterogeneous graph with L_{mac}=2 layers and set the historical context window size to J=3 . The contextual relational penalty is initialized at \alpha_{ctx}=0.1 . For the two-stage asymmetric optimization, we set the valence supervision weight \lambda_{val}=1.0 during the first E_{warm}=5 warm-up epochs. Subsequently, we remove the auxiliary supervision and transition the loss weights to \lambda_{cls}=0.2 and \lambda_{con}=0.8 for topological refinement. We train the network on an NVIDIA RTX 4090 GPU for 15 epochs with early stopping. During training, we use the AdamW optimizer with a learning rate of 5\times 10^{-5} and a batch size of 16. Following previous work [ 8 ] , we adopt Accuracy, Precision, Recall, and F1-score as our evaluation metrics. V Results and Analysis This section presents the main results, ablation studies, qualitative visualizations, and a case study. V-A Main Results Table I evaluates PC-MNet against 25 baselines across seven paradigms on the MUStARD benchmark. PC-MNet achieves a Macro-F1 score of 81.64% and an accuracy of 82.46%, outperforming all baselines. A paired t-test confirms that the 3.14% absolute F1-score improvement over the strongest multimodal baseline, VyAnG-Net [ 20 ] , is statistically significant ( p<0.01 ). Single-modality variants highlight the necessity of multimodal integration: the text-only baseline achieves a 75.09% F1-score, while isolated video and audio models struggle with pragmatic ambiguity, yielding 62.74% and 67.89%, respectively. Integrating these modalities into PC-MNet yields a 6.55% absolute F1-score increase over the text-only model. Consequently, this substantial performance gain demonstrates that explicitly modeling cross-modal incongruity, rather than merely pursuing modality similarity, is essential for robust sarcasm detection. Furthermore, an analysis of the advanced baselines reveals distinct methodological limitations. The recent graph-based model CMGCN [ 16 ] achieves an F1-score of 71.58%. This result indicates that while constructing cross-modal graphs is beneficial, monolithically entangling modalities without explicit polarity-driven routing inevitably leads to semantic over-smoothing. In contrast, the performance of the ESAM [ 33 ] architecture plummets to an F1-score of 69.05%. This sharp performance decline indicates that the associated multi-task framework, which integrates sentiment constraints, remains effective only in scenarios involving short-form multimodal data devoid of temporal context. However, the substantial performance gap between ESAM and PC-MNet demonstrates that soft sentimental regularization is insufficient without structurally decoupling atomic and contextual incongruities. Additionally, large multimodal models such as Llama 3-8B [ 32 ] exhibit a severe precision-recall imbalance on this task, achieving a precision of 67.29% but suffering a significant drop in recall to 51.05%. This significant disparity indicates a reliance on global textual surface cues rather than fine-grained multimodal reasoning. Conversely, PC-MNet maintains a stable balance, achieving 83.61% Macro-Precision and 81.09% Macro-Recall. Consequently, these comparative results confirm that for fine-grained incongruity verification, explicit structural constraints are fundamentally more effective than unconstrained parameter scaling. To further ensure the statistical reliability of these findings, all reported metrics for PC-MNet are averaged over five independent runs with different random seeds. The minimal variance observed rigorously validates that the performance gains stem from the proposed architectural design rather than favorable random initializations, thereby corroborating the aforementioned p<0.01 significance level. TABLE II : Ablation study on the MUStARD benchmark. Model Prec (%) Rec (%) F1 (%) PC-MNet (Full) 83.61 81.09 81.64 w/o Polarity Modulation 77.80 75.30 76.32 w/o \mathbf{e}_{atomic} 78.50 76.10 77.15 w/o \mathbf{e}_{inter} 79.10 76.80 77.80 w/ Tripartite Graph (T-A-V) 78.90 76.40 77.45 w/ Direct Fusion of Branch Topologies 78.10 75.80 76.85 w/o \mathcal{L}_{valence} 79.50 77.10 78.12 w/o \mathcal{L}_{con} 76.80 74.50 75.40 V-B Ablation Study Table II demonstrates PC-MNet performance stems from a decoupled multi-granularity design. All ablated models underperform the best baseline, VyAnG-Net, validating the fairness of the comparative setup. PC-MNet prevents high-dimensional branch-specific composition features ( \mathbf{h}_{T(V)}^{comp} and \mathbf{h}_{T(A)}^{comp} ) from directly entering the final classifier. Bypassing the scalar congruity routing mechanism via direct fusion results in a 4.79% drop in F1-score (to 76.85%). Substituting parallel bipartite-dominant graphs with a single fully connected tripartite graph reduces the F1-score to 77.45%. The degradation demonstrates that projecting unaligned multimodal signals onto a monolithic graph inevitably causes semantic over-smoothing. Replacing polarity-modulated attention with standard dot-product attention causes a 5.32% F1-score drop (to 76.32%). The decrease highlights that explicitly modeling cross-modal contradictions over similarities remains critical for effective sarcasm detection. Removing the micro-level atomic module or inter-sentence contextual GNN drops the F1-score by 4.49% and 3.84%, respectively, validating the necessity of dual-granularity integration. Relying solely on cross-entropy loss causes a 6.24% F1-score degradation. The decline indicates the necessity of a contrastive loss ( \mathcal{L}_{con} ) to explicitly cluster inconsistent topologies. Discarding the initial 5-epoch valence warm-up phase drops F1-score by 3.52%. Lacking continuous valence loss to anchor initial polarity spaces destabilizes the network, hindering subsequent contrastive learning. (a) Without \mathcal{L}_{con} (b) Full PC-MNet Fig. 3 : Visualization of Feature Distributions for Sincere and Sarcastic Samples Without \mathcal{L}_{con} and With Full PC-MNet. (a) standard attention heatmap. (b) Polarity Modulated Attention Heatmap. Fig. 4 : Visualization of Attention Heatmaps for Standard Similarity-Seeking and Polarity-Modulated Contradiction-Seeking Mechanisms. V-C Visualization Discussion Visualizations can validate our architectural design. Figure 3 plots t-SNE projected test representations ( \mathbf{e}_{final} ). Without contrastive loss \mathcal{L}_{con} (Figure 3(a) ), standard optimization causes substantial sarcastic-sincere overlap (0.68 purity). Conversely, the complete PC-MNet (Figure 3(b) ) forms highly separable clusters (0.81 purity). This distinct separation demonstrates contrastive optimization effectively isolates latent sarcastic representations. Figure 4 visualizes cross-attention weights for an utterance pairing a mid-sentence eye-roll (frames F4 and F5) with the positive verbal cue privilege. Standard attention incorrectly highlights smiling frames (F2 and F3). In contrast, polarity-modulated attention accurately localizes contradictory frames. This precise localization confirms that explicitly modeling contradictions captures transient micro-level incongruities essential for robust sarcasm detection. Finally, Figure 5 tracks dynamic routing weights \mathbf{a}_{fuse}=[a_{mic},a_{ctx}]^{\top} . The tracking verifies that the fusion layer adaptively shifts computational focus based on underlying sarcasm mechanisms, avoiding feature redundancy and ensuring optimal dual-level integration. Fig. 5 : Dynamic routing weight distribution ( \mathbf{a}_{fuse} ) across different pragmatic scenarios. (a) Atomic Modulation Weight ( \alpha ). (b) Graph Depth ( L_{mac} ). (c) Contrastive Weight ( \lambda_{con} ). (d) Contrastive Temperature ( \tau ). (e) Contextual Penalty ( \alpha_{ctx} ). (f) Warm-up Epochs ( E_{warm} ). Fig. 6 : Comprehensive hyperparameter sensitivity analysis tracking F1 Score trajectories across three dataset variants. V-D Case Study Success Case A – Modal Incongruity: For an utterance delivered with an eye-roll, the textual sentiment is positive while the visual polarity is negative. Our detection pipeline isolates the facial expression. Because micro-level conflicts dominate, the network appropriately assigns the highest routing weight to the atomic-level representation. Success Case B – Contextual Incongruity: A speaker says, “ I don’t think I’ll be able to stop thinking about it ” in a deadpan tone. Identifying no micro-level conflicts, PC-MNet shifts focus to the inter-sentence GNN, establishing a contradiction edge with the speaker’s historical disinterest. Assigning a contextual weight of 0.85 and a micro weight of 0.15, this dynamic delegation validates the dual-granularity strategy. Error Analysis – Case C: A character mocks a messy room using pseudo-scientific vocabulary in a sincere tone. PC-MNet incorrectly predicts this as non-sarcastic. When sarcasm relies entirely on semantic irony without observable multimodal affective shifts, the model defaults to a literal interpretation. V-E Cross-Dataset Generalization Models exploiting dataset imbalances often suffer severe performance degradation when evaluated on uniform distributions. To formalize the evaluation protocol, the proposed framework and baselines are assessed across the original MUStARD dataset and its two extended datasets (i.e., MUStARD++ and MUStARD++ Balanced ). The purpose is to systematically test architectural robustness against increasing data scale and forced label parity. Table III summarizes the overall generalization trends across these datasets. As demonstrated in Table III , a distinct performance divergence emerges under the rigorous constraints of MUStARD++ Balanced dataset. Specifically, baseline performances decay significantly. For instance, massive unconstrained models such as Llama 3-8B suffer severe degradation, plummeting to F1-scores of 63.80% and 61.10%, respectively. This specific degradation occurs because these baselines rely heavily on spurious correlations and class imbalances within the training data, rather than on fine-grained pragmatic reasoning. Conversely, the performance of PC-MNet remains remarkably stable, even increasing to an 82.45% F1-score on the balanced set. This robust improvement validates a core architectural advantage: the proposed framework relies entirely on a universal, generalized incongruity manifold. Consequently, the PC-MNet architecture effectively resists dataset biases and maintains high detection efficacy across varying distributions. TABLE III : Cross-Dataset Generalization (Macro-F1 %). Bold indicates best; underline indicates second-best. Model MUStARD MUStARD++ MUStARD++ Balanced G 2 SAM 73.50 71.24 70.85 MVIL 75.30 74.15 73.90 ESAM 69.05 65.22 69.05 VyAnG-Net 78.50 76.92 76.10 MO-Sarcation 77.90 77.80 76.55 Llama 3-8B 61.26 63.80 61.10 PC-MNet (Ours) 81.64 81.15 82.45 V-F Hyperparameter Analysis Figure 6 analyzes six critical hyperparameters, confirming the selected configurations avoid overfitting specific data distributions. Detection performance peaks at a micro-atomic modulation weight of \alpha_{mic}=0.5 , proving polarity modulation should enhance, not override, base semantic features. Setting composition graph depth to L_{mac}=2 achieves optimal performance, whereas deeper networks ( L_{mac}\geq 3 ) suffer severe semantic over-smoothing, blurring localized incongruities. Additionally, a contextual penalty of \alpha_{ctx}=0.1 yields peak performance. The specific threshold maintains foundational dialogue context while amplifying necessary pragmatic shifts. Regarding optimization dynamics, Figure 6(c) shows that a contrastive weight of \lambda_{con}=0.8 yields the highest F1-score. The performance peak rigorously validates the asymmetrical loss strategy, prioritizing embedding-space topology structuring. Moreover, a contrastive temperature of \tau=0.07 effectively calibrates the negative penalty during contrastive learning. Finally, Figure 6(f) demonstrates that extending auxiliary valence supervision beyond 5 epochs significantly degrades model performance, dropping the MUStARD++ Balanced dataset F1-score to 66.80%. The substantial degradation suggests prolonged exposure to explicit scalar labels overly restricts the representation space, underscoring the necessity of the proposed two-stage optimization strategy. VI Conclusion We propose PC-MNet, a hierarchical multi-granularity framework for multimodal sarcasm detection explicitly modeling pragmatic incongruities over simple feature fusion. Polarity-modulated attention captures fine-grained cross-modal conflicts, while parallel bipartite-dominant heterogeneous graphs extract intra-utterance structural sarcasm. A strict scalar congruity routing mechanism blocks high-dimensional features from the classifier, routing them as scalar-congruity priors for contextual GNNs and anchors for inconsistency-aware contrastive loss, averting late-fusion gradient interference. Experiments on MUStARD and its balanced datasets demonstrate consistent state-of-the-art performance. Ablations confirm dual-granularity routing and two-stage optimization, utilizing valence warm-up to stabilize representations before contrastive refinement. Relying on observable cues, the model occasionally struggles with dry irony, requiring extensive world knowledge. Future work integrates Multimodal Large Language Models (MLLMs), injecting commonsense priors into nodes, extending to streaming multi-party dialogues. Ultimately, PC-MNet provides a robust paradigm for modeling cross-modal contradiction over mere similarity, advancing broader affective computing tasks.