Generalization Gaps in Political Fake News Detection: An Empirical Study on the LIAR Dataset
S Mahmudul HasanShaily Roy Akib Jawad NafisSyracuse University, Syracuse, NY, USA · 2025
Read Generalization Gaps in Political Fake News Detection: An Empirical Study on the LIAR Dataset by S Mahmudul HasanShaily Roy Akib Jawad NafisSyracuse University, Syracuse, NY, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18533.txt.
Introduction
S Mahmudul Hasan, Shaily Roy, and Akib Jawad Nafis test how far text alone can go in spotting political fake news. Automated fact-checking struggles with subtle, well-written disinformation. Their study, posted to arXiv in December 2025, runs nine machine learning algorithms on the LIAR benchmark, comparing word-count features, TF-IDF, and GloVe embeddings.
They find a hard performance ceiling: fine-grained classification never passes a weighted F1 score of 0.32. A simple linear SVM, at 0.624 accuracy, matches the RoBERTa transformer at 0.620, so model size is not the bottleneck. Tree ensembles memorize the training words, topping 99 percent in training but collapsing to about 25 percent on test data. Adding synthetic data through SMOTE changes nothing. The limit, they conclude, is semantic: without outside knowledge, the words themselves stay ambiguous.
Catalog record
| CREATOR | S Mahmudul HasanShaily Roy Akib Jawad NafisSyracuse University, Syracuse, NY, USA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-18533
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalization Gaps in Political Fake News Detection: An Empirical Study on the LIAR Dataset S Mahmudul HasanShaily Roy Akib Jawad NafisSyracuse University, Syracuse, NY, USA Abstract The proliferation of linguistically subtle political disinformation poses a significant challenge to automated fact-checking systems. Despite increasing emphasis on complex neural architectures, the empirical limits of text-only linguistic modeling remain underexplored. We present a systematic diagnostic evaluation of nine machine learning algorithms on the LIAR benchmark. By isolating lexical features (Bag-of-Words, TF-IDF) and semantic embeddings (GloVe), we uncover a hard “Performance Ceiling,” with fine-grained classification not exceeding a Weighted F1-score of 0.32 across models. Crucially, a simple linear SVM (Accuracy: 0.624) matches the performance of pre-trained Transformers such as RoBERTa (Accuracy: 0.620), suggesting that model capacity is not the primary bottleneck. We further diagnose a massive “Generalization Gap” in tree-based ensembles, which achieve > 99 % >99\% training accuracy but collapse to ≈ 25 % \approx 25\% on test data, indicating reliance on lexical memorization rather than semantic inference. Synthetic data augmentation via SMOTE yields no meaningful gains, confirming that the limitation is semantic (feature ambiguity) rather than distributional. These findings indicate that for political fact-checking, increasing model complexity without incorporating external knowledge yields diminishing returns. Keywords: Fake News Detection, Political News Analysis, Machine Learning, Natural Language Processing, Disinformation Mitigation 1 Introduction Automated fact-checking systems have shown success in detecting sensationalist "fake news," yet they continue to struggle with the nuanced domain of political discourse. Unlike clickbait, which often exhibits structural anomalies or exaggerated language, political disinformation is frequently grammatically and stylistically indistinguishable from accurate reporting. This linguistic overlap raises a fundamental question: can machine learning models infer veracity solely from textual patterns, or is external context necessary? The release of the LIAR benchmark dataset wang2017liar provided a standardized testbed for this question, yet reported baselines remain consistently low. Wang et al. wang2017liar reported a fine-grained test accuracy of 27.4% using hybrid CNN models, and subsequent studies confirm the intrinsic difficulty of the task. Even large pretrained language models such as RoBERTa achieve only moderate performance on simplified binary variants, underscoring the limited veracity signal present in short political statements khan2021benchmark . We hypothesize that this persistent performance ceiling arises from two interacting factors: shortcut learning geirhos2020shortcut , wherein models memorize superficial lexical cues rather than veracity-related patterns, and semantic ambiguity , which prevents reliable discrimination between degrees of truth without external evidence. To test this hypothesis, we conduct a comparative analysis across a diverse set of classification algorithms, including linear models (SVM, Logistic Regression) and high-variance ensemble methods (Random Forest, Extra Trees, XGBoost). To isolate the role of feature representation, we evaluate each model using Bag-of-Words, TF-IDF, and static word embeddings (GloVe). Our results reveal a consistent empirical performance ceiling for text-only approaches: across models and representations, fine-grained classification does not exceed 0.32 Weighted F1 , while binary classification plateaus at approximately 0.64 Weighted F1 . Although tree-based ensemble methods achieve near-perfect training accuracy ( >99\% ), they exhibit severe degradation on held-out test data, indicating reliance on lexical memorization rather than robust semantic inference zhu2022generalizing . The primary contributions of this work are as follows: • Establishing an Empirical Performance Ceiling: We provide a systematic evaluation of nine machine learning algorithms on the LIAR dataset, demonstrating that text-only models consistently plateau at approximately 0.32 Weighted F1 on fine-grained tasks. • Characterizing the Generalization Gap: We quantify the extent of overfitting in political text classification, showing that tree-based ensemble models exhibit large generalization gaps ( \Delta\approx 0.75 ), indicative of shortcut learning. • Evaluating the Role of Class Imbalance: Through synthetic oversampling with SMOTE, we show that class imbalance alone does not explain poor performance, suggesting that semantic ambiguity is a primary limiting factor for text-only approaches. 2 Related Work The rapid growth of user-generated content on social media platforms has intensified concerns around information integrity, motivating extensive research into automated fact-checking systems horne2017just ; van2020you . While early verification efforts relied on manual analysis by professional journalists, the scale of online content has driven the development of computational approaches zhang2020overview . Prior work broadly spans linguistic feature-based models and hybrid deep learning frameworks that incorporate auxiliary signals, yet the empirical limits of text-only fact-checking remain insufficiently characterized. 2.1 Linguistic Benchmarks and the LIAR Challenge Early studies in misinformation detection focused on surface-level linguistic cues and syntactic features. Aldwairi and Alwahedi aldwairi2018detecting demonstrated high accuracy in clickbait detection using syntactic analysis of headlines, while Reis et al. reis2019supervised reported strong performance with psycholinguistic features on curated news datasets. However, these tasks differ substantially from political fact-checking, where deceptive statements often adhere to professional journalistic norms. This challenge was formalized with the introduction of the LIAR dataset by Wang et al. wang2017liar , who reported a fine-grained classification accuracy of only 27.4\% using hybrid CNN models. Subsequent evaluations khan2021benchmark confirmed that even large pretrained language models, including BERT-based architectures, struggle to generalize on fine-grained political statements when relying solely on textual input. These findings suggest that linguistic cues alone provide limited discriminative power in this domain. 2.2 Deep Learning and Hybrid Frameworks To address these limitations, later work incorporated metadata and contextual signals beyond the statement text. Ruchansky et al. ruchansky2017csi proposed the CSI framework, combining textual content with user responses and source credibility, while Karimi et al. karimi2018multi explored multi-class deception modeling using additional contextual features. Although such hybrid approaches improve performance, they conflate linguistic modeling with external signals, making it difficult to isolate the capabilities and limitations of text-only representations. 2.3 Generalization Gap and Shortcut Learning Recent studies have highlighted that neural models often exploit spurious correlations rather than learning task-relevant semantics. Geirhos et al. geirhos2020shortcut characterize this phenomenon as shortcut learning , where models achieve high training performance without solving the underlying problem. In the context of political fact-checking, Zhu et al. zhu2022generalizing identified entity bias , showing that classifiers frequently memorize speaker-specific truthfulness patterns instead of analyzing statement content. Such biases contribute to a pronounced generalization gap , wherein models perform well on seen data but fail to generalize to unseen entities or contexts. Despite extensive work on both linguistic and hybrid systems, there remains a lack of systematic analysis that isolates the generalization limits of text-only models across algorithmic complexity and feature representations. We address this gap by quantifying generalization behavior across a spectrum of traditional classifiers. Our findings provide empirical evidence that linguistic patterns alone offer limited signal for robust political fact-checking, supporting recent efforts that integrate external knowledge and contextual reasoning pan2018content ; jin2025dynamic . LIAR Dataset Valid Train Test Class Balancing (SMOTE) Preprocessing Feature Extraction Semantic (GloVe) Lexical (BoW/TF-IDF) Model Training Evaluation Figure 1 : Schematic of the experimental framework comparing lexical (TF-IDF) and semantic (GloVe) feature representations across linear and ensemble models to quantify the generalization gap. 3 Methodology and Experimental Design To rigorously evaluate the limitations of traditional machine learning in political fact-checking, we implemented a systematic benchmarking pipeline. Our methodology is designed to isolate the Generalization Gap by contrasting high-capacity ensemble methods against low-variance linear baselines across varied feature representations. This section details the dataset characteristics, feature engineering strategies, model architectures, and experimental protocols. Figure 1 illustrates the end-to-end experimental pipeline. 3.1 Dataset