How do Language Models Generate Slang: A Systematic Comparison between Human and Machine-Generated Slang Usages

Siyang Wu 1, 2 , Zhewei Sun 2 1 Data Science Institute, University of Chicago, Chicago, Illinois 2 Toyota Technological Institute at Chicago, Chicago, Illinois [email protected], [email protected] · 2025

Read How do Language Models Generate Slang: A Systematic Comparison between Human and Machine-Generated Slang Usages by Siyang Wu 1, 2 , Zhewei Sun 2 1 Data Science Institute, University of Chicago, Chicago, Illinois 2 Toyota Technological Institute at Chicago, Chicago, Illinois [email protected], [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15518.txt.

Introduction

Siyang Wu and Zhewei Sun compare slang made by people with slang made by language models. Slang is informal speech that machines find hard. Models are now used to spot and explain slang, so their grasp of it matters. The pair ask whether model made slang matches real human use in style and craft.

Their frame checks three things: biases in how machines view slang, craft in new and reused coinages, and how useful model slang is as teaching examples. They compare a human slang dictionary with output from two known models. The models show clear biases in how they see slang. Still, they have caught much of slang's creative side.

Catalog record

CREATORSiyang Wu 1, 2 , Zhewei Sun 2 1 Data Science Institute, University of Chicago, Chicago, Illinois 2 Toyota Technological Institute at Chicago, Chicago, Illinois [email protected], [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15518

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How do Language Models Generate Slang: A Systematic Comparison between Human and Machine-Generated Slang Usages Siyang Wu 1, 2 , Zhewei Sun 2 1 Data Science Institute, University of Chicago, Chicago, Illinois 2 Toyota Technological Institute at Chicago, Chicago, Illinois [email protected], [email protected] Abstract Slang is a commonly used type of informal language that poses a daunting challenge to NLP systems. Recent advances in large language models (LLMs), however, have made the problem more approachable. While LLM agents are becoming more widely applied to intermediary tasks such as slang detection and slang interpretation, their generalizability and reliability are heavily dependent on whether these models have captured structural knowledge about slang that align well with human attested slang usages. To answer this question, we contribute a systematic comparison between human and machine-generated slang usages. Our evaluative framework focuses on three core aspects: 1) Characteristics of the usages that reflect systematic biases in how machines perceive slang, 2) Creativity reflected by both lexical coinages and word reuses employed by the slang usages, and 3) Informativeness of the slang usages when used as gold-standard examples for model distillation. By comparing human-attested slang usages from the Online Slang Dictionary (OSD) and slang generated by GPT-4o and Llama-3, we find significant biases in how LLMs perceive slang. Our results suggest that while LLMs have captured significant knowledge about the creative aspects of slang, such knowledge does not align with humans sufficiently to enable LLMs for extrapolative tasks such as linguistic analyses. 1 Introduction Slang is a type of informal language that is commonly used in colloquial speech Sornig 1981 . The use of slang is both creative Warren 1992 ; Eble 2012 and ephemeral Eble 1989 , meaning that slang is not only more difficult to comprehend compared to conventional language, but it is also necessary to handle an ever-evolving repertoire of novel slang usages. Such characteristics of slang necessitate natural language processing (NLP) systems that can adapt to unseen slang usages without expansive retraining. While earlier work on NLP for slang often resorts to retrieval-based systems that can hardly generalize (e.g., Pal and Saha 2013 ), recent work has been successful in developing systems for the automatic detection Pei et al. 2019 ; Liu and Seki 2021 ; Sun et al. 2024 , generation Kulkarni and Wang 2018 ; Sun et al. 2021 , and interpretation Ni and Wang 2017 ; Sun et al. 2022 ; Mei et al. 2024 ; Wuraola et al. 2024 of slang that can generalize well toward novel slang usages that have never been seen during training. In particular, large language models (LLMs) have been very effective in many tasks involving slang under both zero-shot and few-shot settings, suggesting that the LLMs have, to some extent, captured structural knowledge about slang that enables generalization. Figure 1: Our evaluative framework considers three core aspects of knowledge: 1) Characteristics, 2) Creativity, and 3) Informativeness by comparing human and machine-generated slang usages. The + sign indicates fine-tuning. It is not well understood, however, what underlying structures about slang the LLMs have captured and whether they are comparable to human knowledge. Such insights are valuable in gauging the reliability of downstream applications that require a precise characterization of slang. For example, a model with misaligned knowledge would be systematically biased toward detecting certain types of slang usages. Slang usages generated by such models can also misinform downstream agents that consume the content (e.g., model distillation), as well as linguistic analyses that rely on LLMs for automatic annotation. We make a first step toward the interpretation of LLMs’ internal knowledge about slang by collecting a dataset of machine-generated slang usages from GPT-4o OpenAI 2024 and Llama-3 AI@Meta 2024 that span a diverse set of controlled conditions. Specifically, we prompt the LLMs to generate novel slang usages that each encompasses 1) A slang term, 2) A sense definition sentence, and 3) A usage context. We perform both controlled generation where the model is provided with existing senses from a slang dictionary and an uncontrolled setting where the model is allowed to generate slang usages attached to any senses. Under each condition, we further constrain the type of word choices that can be made. This includes 1) Lexical coinage where the the model is prompted to create a new term, 2) Word reuse where the model chooses an existing term in the lexicon, and 3) Free-form generation without any restrictions. We collect at least 1,000 valid generations in each setting for a total of 58,197 machine-generated slang usages 1 1 1 Code and data available at: https://github.com/siyangwu1/LLM-Slang-Dictionary . Using this dataset, we make a systematic comparison between human-generated slang usages attested by slang dictionaries and machine-generated slang usages. Illustrated in Figure 1 , we propose an evaluative framework that makes comparisons along three core aspects: 1) Our framework compares the aggregate usage characteristics of the generated slang usages to discern any systematic biases in how machines perceive slang; 2) It measures and compares the creativity of slang usages, examining morphological complexity and semantic coherence of coined terms as well as semantic novelty and contextual surprisal for cases of reuse; 3) Model distillation is performed using both human and machine-generated slang usages as examples to measure their informativeness toward a diverse set of NLP tasks for slang. Our results show that while LLMs have captured sufficient creative knowledge to generate plausible slang usages, the generated usages still deviate significantly from human-generated usages in certain aspects. We make the following contributions in this paper: 1) A dataset of slang usages generated by GPT-4o and Llama-3 that enables studies of machine-generated informal language; 2) An evaluative framework that assesses knowledge alignment between human and machine-generated language use; 3) The first systematic comparison between human and machine-generated slang usages. 2 Related Work 2.1 Knowledge-driven processing of slang Earlier work relies on building and retrieving from high quality data sources to enable machine processing of slang Pal and Saha 2013 ; Dhuliawala et al. 2016 ; Gupta et al. 2019 . Such approaches require constant updates to obtain new knowledge and thus cannot be efficiently combined with large machine learning models that are expansive to retrain. Meanwhile, encoder models such as BERT Devlin et al. 2019 perform poorly on slang due to limited scale at the time. To address this, a series of work has been proposed to inject linguistic knowledge about slang into the models to create inductive biases that enable efficient learning. Such an approach has been successfully applied to enable automatic detection Pei et al. 2019 ; Liu and Seki 2021 , generation Kulkarni and Wang 2018 ; Sun et al. 2019 ; Sun et al. 2021 , and interpretation Sun et al. 2022 of slang usages that have not been seen during training. Most related to our work is Kulkarni and Wang 2018 who built generative neural models of slang word coinage for common types of word formation strategies (e.g., blending) employed in slang word formation identified by prior linguistic research Mattiello 2013 . Also, Sun et al. 2021 studied cases of word reuse in slang by modeling slang generation as a sense extension phenomenon. In their work, contrastive learning Baldi and Chauvin 1993 ; Bromley et al.

Read the complete text