Identifying Emerging Concepts in Large Corpora

Sibo Ma, Julian Nyarko, [email protected] · 2025

Read Identifying Emerging Concepts in Large Corpora by Sibo Ma, Julian Nyarko, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-21315.txt.

Introduction

Ma and Nyarko spot emerging concepts in huge text piles by watching embedding heat maps shift in twenty twenty-five. New ideas light up fresh zones in the map of word vectors. Their method catches them soon after birth with high accuracy. It beat common rival methods in their tests.

United States Senate speeches from nineteen forty-one to twenty fifteen served as the field trial. The minority party proved the more active source of new concepts. Some concepts tracked tightly with senators' race, ethnicity, and gender. Working code is public. The tool suits any large, dated text hoard.

Catalog record

CREATORSibo Ma, Julian Nyarko, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-21315

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Identifying Emerging Concepts in Large Corpora Sibo Ma, Julian Nyarko, [email protected] Abstract We introduce a new method to identify emerging concepts in large text corpora. By analyzing changes in the heatmaps of the underlying embedding space, we are able to detect these concepts with high accuracy shortly after they originate, in turn outperforming common alternatives. We further demonstrate the utility of our approach by analyzing speeches in the U.S. Senate from 1941 to 2015. Our results suggest that the minority party is more active in introducing new concepts into the Senate discourse. We also identify specific concepts that closely correlate with the Senators’ racial, ethnic, and gender identities. An implementation of our method is publicly available. 1 Introduction The identification of new ideas and concepts within large corpora is of core interest both in computational linguistics, the social sciences, and the humanities. For instance, Hofstra et al. (2020) discover new innovations in a corpus of scientific articles. They find that minorities often introduce novel contributions to the scientific discourse, but those innovations are disproportionately discounted. Charlesworth et al. (2022) examine how new stereotypes towards ethnic and racial minorities evolved during the 19 th and 20 th centuries. And Hanley et al. (2024) identify misinformation at its conception and track its spread and influence on public discourse. Recent advances in natural language processing offer more robust alternatives. Transformer-based models, such as BERT ( Devlin et al., 2019 ) and T5 Raffel et al. (2020) , leverage contextual embeddings to encode semantic meaning beyond individual words, making them well-suited for tracking emerging concepts that lack stable lexical forms. However, most existing methodologies for text analysis are not specifically designed for identifying emergent concepts. Instead, they often apply general-purpose techniques that do not account for the distinct temporal patterns associated with conceptual emergence. As a consequence, existing approaches often lack sensitivity, allowing for the accurate identification of emerging concepts only after these concepts have become well-represented in the underlying corpora. To the extent that methods exist that are specifically targeted at emerging concept identification, they tend to be supervised Charlesworth et al. (2022) ; Kulkarni et al. (2015) , thus requiring the investigator to know ex ante what new concepts to look for. They also tend to operate at the word-level, thus making it impossible to identify concepts that are not easily representable using a distinct unigram or bigram. But consistent with the semantic perspective, we understand concepts as abstract objects—propositions that can be expressed in a multitude of ways Margolis and Laurence (2007) . While some concepts can be captured by individual words or phrases (e.g., “climate change”), we assume that many concepts are more complex and cannot be reduced to a single lexical unit (e.g., “negative sentiment towards the Affordable Care Act as a form of governmental invasion” Fisher and Larsen (2019) ). This in turn requires a methodological approach that goes beyond the word-level and is able to capture the broader semantic structures that define emerging concepts. In this paper, we introduce a novel methodology to identify emerging concepts in large text corpora that we also make publicly available. 1 1 1 An implementation can be found at https://github.com/Crabtain959/new_concept_detection . Intuitively, our methodology leverages heatmaps of the embedding space to identify those regions that are subject to sudden, long-lasting increases in density. Because we benchmark the density increase against commonly observed, non-systematic changes, our method performs particularly well at identifying emerging concepts shortly after their inception, well before they become broadly represented in the underlying corpus. Our method operates at the sentence level (or any other, larger textual unit), thus allowing for identification of complex concepts that cannot easily be captured with a distinct word or phrase. In several evaluations, we demonstrate the performance of our proposed methodology, and compare it to other approaches that have been employed to identify emerging concepts in text. In a last step, we illustrate the utility of our method for the social sciences and digital humanities by exploring the introduction of new concepts in the U.S. Senate debates from 1941 (77 th Congress) to 2015 (114 th Congress). At a macro level, we find a consistent pattern showing that the minority party introduces new concepts and ideas with greater frequency than the majority. This finding lends support to claims by other scholars about strategically different behavior of the minority party in legislative bodies Jenkins et al. (2023) ; BALLARD and CURRY (2021) , including their way to converse Pozen et al. (2019) . 2 Related Work Perhaps closest in spirit to our motivation is a recent study by Vicinanza et al. (2022) . They identify novel ideas using word-level perplexity, such that sentences with unexpected word combinations are deemed to reflect novelty. They provide convincing evidence that their approach identifies novel ideas at a macro level. In their approach, syntactic idiosyncracies are receiving the same weight as unusual semantic occurrences. And since it operates at the word level, there are likely limits to the linguistic complexity with which new ideas can be identified. In contrast, our approach centers around textual embeddings, a literature with a long tradition. A number of early contributions interested in the identification of new concepts characterized the problem in terms of diachronic meaning shift detection. That is, a new concept would be identified by the emergence of a new word sense for a given word. Notably, Hamilton et al. (2016) demonstrated that word meaning evolution follows predictable trajectories, influenced by frequency and semantic drift, highlighting the importance of tracking concept emergence over time. Naturally, the associated methodologies are centered around the use of word embedding models. Several studies have explored different approaches to detecting semantic change, including cosine distance between word embeddings ( Del Tredici et al., 2019 ; Kulkarni et al., 2015 ; Shoemark et al., 2019 ) , Bayesian models for temporal word representations ( Frermann and Lapata, 2016 ) , and topic modeling techniques such as Latent Dirichlet Allocation (LDA) ( Lau et al., 2012 ; Blei et al., 2001 ) . Other refinements incorporate contextual embeddings ( Martinc et al., 2020 ) or focus on modeling semantic drift in historical corpora ( Periti et al., 2024 ; Boholm et al., 2024 ) . Surveys such as Periti and Montanelli (2024) provide an overview of these methodologies and their relative strengths and limitations. Despite their effectiveness in tracking known shifts, these methods face two key limitations. First, since they analyze shifts at the word-level, they are not well-suited to detect emerging concepts that cannot easily be associated with a single word or phrase ( Stewart and Eisenstein, 2018 ; Hofmann et al., 2020 ) . Second, these methods are supervised in the sense that the investigator needs to predefine the list of words for which a new meaning may emerge. In that way, they are better suited to identify when a known semantic shift has occurred. However, they are not aimed at identifying previously unknown, emergent concepts. More recent approaches have relied on the detection of concepts by applying clustering algorithms to text embeddings Sia et al. (2020) ; Giulianelli et al. (2020) ; Angelov (2020) ; Grootendorst (2022) . These unsupervised methods do not require predefined

Read the complete text