From Ground Trust † to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse

Seunguk Yu, Jungmin Yun, Jinhee Jang, Youngbin Kim · 2025

Read From Ground Trust † to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse by Seunguk Yu, Jungmin Yun, Jinhee Jang, Youngbin Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14712.txt.

Introduction

Seunguk Yu and three co-authors study offensive language judgement in 2025 on contemporary Korean political discourse. Offensive language shifts quickly, yet detectors are often trained and tested on outdated data. Generalisation to unseen current text is rarely measured, especially for Korean.

A large new data set of current political text anchors the study. Three refined judgement methods, each a representative detection approach run under strong conditions, label the data without a ground truth. Agreement patterns are probed with a leave one out strategy, and pseudo labels serve as a stand in truth for scoring. Disparities among methods show how fragile a single detector verdict is on fast moving political speech.

Catalog record

CREATORSeunguk Yu, Jungmin Yun, Jinhee Jang, Youngbin Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-14712

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Ground Trust † to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discourse Seunguk Yu, Jungmin Yun, Jinhee Jang, Youngbin Kim Abstract Warning: this paper contains expressions that may offend the readers. Although offensive language continually evolves over time, even recent studies using LLMs have predominantly relied on outdated datasets and rarely evaluated the generalization ability on unseen texts. In this study, we constructed a large-scale dataset of contemporary political discourse and employed three refined judgments in the absence of ground truth. Each judgment reflects a representative offensive language detection method and is carefully designed for optimal conditions. We identified distinct patterns for each judgment and demonstrated tendencies of label agreement using a leave-one-out strategy. By establishing pseudo-labels as ground trust † for quantitative performance assessment, we observed that a strategically designed single prompting achieves comparable performance to more resource-intensive methods. This suggests a feasible approach applicable in real-world settings with inherent constraints. 1 Introduction Offensive language has emerged as a persistent linguistic issue in online discourse, broadly encompassing expressions intended to insult, demean, or ridicule others 1 1 1 This study examines a broader spectrum of offensive language , including hate speech —expressions that promote hatred toward specific individuals or groups. Mnassri et al. 2024 ; Pradhan et al. 2020 . This concern is particularly evident in social media comments, where users articulate and exchange diverse opinions Abdelsamie et al. 2024 ; Aklouche et al. 2024 . The form and content of such language frequently depend on the underlying factual context Ghenai et al. 2025 , and they are recognized as key factors in intensifying social tensions and driving the polarization of public opinions Vasist et al. 2024 ; Kaur et al. 2024 . In this study, we focus on Korean online news platforms as a forum for public discourse in the context of rapidly shifting political dynamics Jin 2025 . Most prior research on offensive language in Korean has relied on outdated datasets, with the latest collected in early 2022, limiting their ability to capture recent political developments and emerging patterns of social conflict Park et al. 2024 ; Park et al. 2023a ; Park et al. 2023b ; Jeong et al. 2022 ; Lee et al. 2022 ; Kang et al. 2022 . To advance the field, we constructed a new dataset by curating political news articles and user comments posted throughout 2024 on the most widely used news platform, ensuring it reflects the evolving sociopolitical landscape Earle and Hodson 2022 ; Kleinnijenhuis et al. 2019 . The news articles are categorized into six topics: Presidential Office , National Assembly / Political Parties , North Korea , Administration , National Defense / Foreign Affairs , and General Politics . Through the fine-grained framework, we constructed the PoliticalK.O dataset ( Political comments for K orean O ffensive language), comprising 114,000 articles and 9.28 million user comments. Since the collected comments contained no ground truth for offensiveness, we employed a diverse set of established offensive language detection methods within a carefully designed framework to assign predicted labels to each comment. Figure 1: We adopt three distinct judgments for offensive language detection: supervised ensemble judgment ( SEJ ) on the left, prompt-variants ensemble judgments ( PEJ ) in the center, and multi-debate reasoning judgment ( MRJ ) on the right. Each constitutes a distinct approach to label inference, characterized by its methodology for decision guidance and aggregation, tailored to the newly collected comments from the PoliticalK.O . We introduce three main judgments 2 2 2 Conventional classification tasks evaluate how well predictions align with ground truth, but in our case, we refer to the outcomes of each method as judgment due to the absence of ground truth for the unseen comments. : supervised ensemble judgment ( SEJ ), prompt-variants ensemble judgment ( PEJ ), and multi-debate reasoning judgment ( MRJ ), which are outlined in Figure 1 . First, SEJ utilizes five existing offensive language datasets to fine-tune PLMs to an optimal configuration, combining their predictions through majority voting. Despite the datasets being outdated, we intended to maximize the utilization of meticulously curated data constructed through human annotation Pandey et al. 2022 ; Kwon et al. 2022 . Following this, PEJ employs three LLMs with the five prompt variants, leveraging contextual information such as explicit definitions of offensive language or article summaries, which are then aggregated by majority voting. We aimed to fully harness the relevant information from an in-context learning perspective Sun et al. 2023 ; Dong et al. 2022 ; Brown et al. 2020 . Lastly, MRJ assumes that each comment can be interpreted as offensive or non-offensive depending on the perspective, and determines the final label through a multi-agent debate. We intended to enhance the LLM’s decision-making capabilities by leveraging contrastive stances Park et al. 2024 ; Du et al. 2023 . We finally derived offensive labels for the unseen comments across three judgments with optimized configurations. Although numerous studies have explored offensive language detection, their applicability to emerging data remains underexplored . To fill this gap, we conducted a large-scale analysis on the PoliticalK.O by implementing judgments tailored to the current sociolinguistic landscape. This framework enabled us to examine offensive tendencies of each judgment, identify potential shared decision criteria, and assess label consistency when certain judgments were excluded. Through this analysis, we obtained a granular understanding of how offensive language is perceived in unseen comments from diverse perspectives. Furthermore, in the absence of ground truth labels for the newly collected comments, we evaluated performance of each judgment through pseudo-labeling Ahmed et al. 2024 ; Yang et al. 2023a ; Zou and Caragea 2023 . By treating the aggregated judgments as a ground trust † , we conducted an analytic evaluation of the three judgments and their respective components. We then examined how combinations of prior datasets and prompt variants yielded reliable offensive language detection results by leveraging proxy ground trust † even without human annotations. Our analysis revealed practical scenarios where a single prompt achieves comparable performance to more resource-intensive methods, highlighting practical approaches for real-world use cases. To facilitate future research in this aspect, we release our dataset and the labels generated by each judgment as open-source resources 3 3 3 https://github.com/seungukyu/PoliticalK.O . 2 Related Work 2.1 Recent Advances in Offensive Language Detection BERT brought significant advances to offensive language detection by enabling bidirectional context modeling Althobaiti 2022 ; Roy et al. 2022 ; Caselli et al. 2021 . This capability allowed models to capture context-sensitive features such as indirect hostility and metaphorical expressions that earlier machine learning approaches struggled to address Ramos et al. 2024 ; Xiao et al. 2024a . Concurrently, researchers began developing more fine-grained and context-aware datasets to account for sociocultural factors Din et al. 2025 ; Pachinger et al. 2024 ; Deng et al. 2022 ; Mathew et al. 2021 ; Rosenthal et al. 2021 ; Zampieri et al. 2019 . These efforts introduced richer taxonomies and labeling schemes tailored to specific communities. Since the emergence of LLMs, research has adopted prompt-based approaches such as providing models with explicit criteria for identifying offensiveness Lu et al. 2025 ; Nghiem and Daumé Iii 2024 , or employing chain-of-thought

Read the complete text