mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection
Dominik Macko · 2026
Read mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection by Dominik Macko (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02712.txt.
Introduction
Dominik Macko reports a single-author system entry for a shared task on conspiracy detection in social media comments. The task is binary: does a Reddit comment express a conspiracy belief? Training data were limited, which shapes the whole approach. The entry builds on methods first developed for detecting machine-generated text.
To cope with scarce data, the system uses data augmentation and self-training, then fine-tunes a large open language model for classification. The result was competitive, placing in the eighty-fifth percentile among fifty-two submissions. Macko presents the outcome as evidence that a pipeline built for one text classification problem can transfer to another with similar traits. The paper is a compact system description rather than a new theory.
Catalog record
| CREATOR | Dominik Macko |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.02712
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection Dominik Macko Abstract SemEval-2026 Task 10 is focused on conspiracy detection. Specifically, the goal is to detect whether a Reddit comment expresses a conspiracy belief. Our submitted mdok-style system utilizes data augmentation and self-training (to cope with a rather small amount of training data) to finetune the Qwen3-32B model for a binary text-classification task. The submitted system is very competitive, ranking in the 85th percentile (8th out of 52 submissions). The results shown that our approach, which originated in machine-generated text detection, can be used for conspiracy detection as well. 1 Introduction The Psycholinguistic Conspiracy Marker Extraction and Detection (PsyCoMark) shared task (SemEval-2026 Task 10) combines psychology and natural language processing disciplines to shed light on how conspiracy theories are expressed in social media (Reddit) conversations ( 10 ) . It includes two subtasks, while we have focused only on subtask 2 that is dedicated to binary detection of conspiracy belief in a textual comment. Specifically, the goal is to classify English Reddit comments as either conspiracy-related or not (in a Yes/No manner). The training data contains 1,715 positive samples and 2,263 negative samples. The texts have been filtered to a length of 160 to 1,000 characters. The proposed system is heavily based on mdok ( m achine d etector o f K InIT) ( 9 ) , a robust detector of machine-generated text winning a shared task of PAN@CLEF2025 ( 2 ) . Our final submitted system ( mdok-style conspiracy detector ) 1 1 1 https://github.com/kinit-sk/mdok-style-psycomark2026 utilizes data augmentation and self-training to cope with a rather small amount of training data. It is based on the Qwen3-32B model ( 14 ) , finetuned using QLoRA parameter-efficient finetuning technique (PEFT) for a binary sequence classification task. 2 Related Works and Background Conspiracy detection is part of the broader field of misinformation detection. Early approaches framed the task as a text classification problem using traditional machine learning and NLP techniques, relying on lexical and stylistic features with classification models, such as SVM (support vector machines) ( 3 ) . Subsequent research introduced psycho-linguistic and emotional features, showing that conspiracy-related narratives exhibit distinct linguistic patterns such as heightened certainty, emotional intensity, and distrust framing. Incorporating such signals improves detection performance beyond surface-level textual representations ( 5 ) . More recently, large language models (LLMs) have been applied to conspiracy detection tasks due to their strong contextual understanding. While LLMs improve generalization, they may suffer from hallucination and precision issues. Emotion-aware finetuned LLM frameworks have been proposed to integrate affective signals into transformer-based architectures, achieving improved performance over vanilla LLM baselines ( 6 ) . For the task of binary conspiracy detection within a given text, we have utilized our experience in a similar task focused on machine-generated text detection. We have transferred our best approach to this kind of a different problem. We have already started to experiment with finetuning LLMs for a binary classification task at SemEval-2024 Task 8 ( 12 ) , where we have observed a better text-classification performance of a finetuned LLM (size of 7B parameters) in comparison to the traditionally used small pre-trained (BERT-like) models. We have further enhanced the robustness of the finetuning process ( 7 ) , which has finally resulted in the mdok ( 9 ) finetuning approach. mdok has ranked 1st in both subtasks of PAN@CLEF2025 ( 2 ) , one of which was focused on robust binary text classification (specifically, the detection of machine-generated texts). In this shared task of SemEval-2026 Task 10, we utilize the mdok’s obfuscation and anonymization of training data as data augmentation techniques and combine them with self-training to cope with a small amount of labeled data. Self-training ( 1 ) is an iterative semi-supervised learning technique, where a model acts as its own teacher to leverage unlabeled data. Firstly, a so-called teacher model is trained on a small amount of data with available “golden” labels (often labeled by humans). The weakly trained model is then used to predict “silver” labels (a.k.a., weak labels or pseudo labels, i.e., they may contain erroneously labeled samples) using any unlabeled data. Since such labels might be inaccurate, usually only the most confident predictions are used to retrain the model. This process can repeat, allowing to continuously expand the training set. However, the errors from the first iteration of silver labels can propagate throughout the learning. 3 System Overview As mentioned above, the PsyCoMark shared task contains rather small amount of train data (at least for bigger LLMs). To cope with the problem we are proposing a combination of data-augmentation techniques as well as using self-training to incrementally label the provided unlabeled dev and test sets. 3.1 Data Augmentation In our mdok-style approach for binary conspiracy detection, we use four data-augmentation techniques described below. To use each technique, we have copied the training texts that have been afterwards modified by the corresponding technique. In order to reduce prevalence of augmented texts (in comparison to original texts), we have used only 10% out of each technique to combine into the training. De-duplication of the final train set removes the redundant texts (i.e., unmodified copies). Anonymization. The orignal PsyCoMark data have already replaced URLs in the texts by a tag of [URL]. Similarly, we have used a text-prepocessing procedure (available in the original mdok) to replace the regex-identified email addresses, user mentions, and phone numbers by the tags of [EMAIL], [USER], [PHONE]. In comparison to the original mdok, we are not using this anonymization procedure for preprocessing all the texts, but for modification of copied data for data augmentation. Lower-casing and Upper-casing. We consider conspiracy beliefs to be invariant on the casing of the text (especially in social media, often containing informal style). Therefore, to make the detection more focused on meaning rather than on the visual form, we integrate both, lowercase and uppercase copies of the original texts. This augmentation not only increases the number of training samples, but it also makes the detection case insensitive. Homoglyphication. Homoglyph attacks (swapping visually similar characters of different scripts) are quite successful in confusing text classifiers (dependent on internal representation of characters and words, such as in tokenization). In our previous work on machine-generated text detection ( 8 ) , we have found out that this confusion effect on classifiers can be effectively dealt with by inclusion of homoglyphied samples during training. As an data augmentation, we have used such homoglyphication of copied texts, increasing the train-set size while also making the conspiracy detector more robust. 3.2 Self-Training In the first round, the selected LLM has been trained purely on the provided train set. Afterwards, it provided predictions of labels of the dev and test sets, while for each prediction we have dumped also a probability of the positive class (“Yes”). Based on such probabilities, we have kept as silver labels only those that had very high probability ( \geq 0.99) as positive and those that had very low probability ( \leq 0.01) as negative (in order to minimize propagation of errors). Such silver-labeled samples have been afterwards combined with the training data for retraining the detector. 3.3 Finetuning Process Since the provided dev split of