Word Sense Detection Leveraging Maximum Mean Discrepancy

Kensuke Mitsuzawa · 2025

Read Word Sense Detection Leveraging Maximum Mean Discrepancy by Kensuke Mitsuzawa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01602.txt.

Introduction

Kensuke Mitsuzawa, writing alone in a 2025 paper, proposes a way to spot words that are changing meaning. Word senses shift over time, and tracking those shifts helps read old texts and social change. His method borrows a statistics tool, maximum mean discrepancy, which measures how far apart two sets of data sit.

The tool picks out the features that best mark a word's use in each era and scores how much that use has moved. That gives both a flag for shifting words and an account of how the shift unfolded over several periods. The author says this is the first use of the tool for sense change, and tests support the method. Such tools can help dictionary makers and historians date when new senses took hold.

Catalog record

CREATORKensuke Mitsuzawa
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01602

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Word Sense Detection Leveraging Maximum Mean Discrepancy Kensuke Mitsuzawa Abstract Word sense analysis is an essential analysis work for interpreting the linguistic and social backgrounds. The word sense change detection is a task of identifying and interpreting shifts in word meanings over time. This paper proposes MMD-Sense-Analysis , a novel approach that leverages Maximum Mean Discrepancy (MMD) to select semantically meaningful variables and quantify changes across time periods. This method enables both the identification of words undergoing sense shifts and the explanation of their evolution over multiple historical periods. To my knowledge, this is the first application of MMD to word sense change detection. Empirical assessment results demonstrate the effectiveness of the proposed approach. 1 Introduction Natural language has been developed and changed over time, and, as time goes on, the sense of vocabulary has often changed. For example, the word gay had the sense of carefree and has shifted to homosexual during the 20th century ( kutuzov-2018 ) . In the Natural Language Processing (NLP) community, the task of word sense change analysis detects these changes in vocabulary senses over time. The task is also called “semantic change”, “semantic shift”, “temporal embeddings”, “diachronic embeddings”, or “lexical semantic change” Periti-2024-lexical-semantic-change , however, in this paper, I call it a “sense change detection” task, following the terminology ( kutuzov-2018 ) . The application of the sense change detection is, for instance, linguistic and social background analysis. By examining vocabulary that changes over time, we can gain insights into the social backgrounds of the speakers who use those words. For instance, the study timmermans-2022 investigates sense change detection in a collection of Dutch texts from the 18th and 19th centuries, and infers the historical social context in which nationalism emerged in the Netherlands. The two major goals of the sense change detection are identifying words that experience the meaning shift between two time periods and explaining the sense change of words over multiple periods kiyama-2025 . To achieve these goals, recent work employs vector representations trained by word embedding models Aida-2021 or large language models Lazaridou-2021 ; Loureiro-etal-2022-timelms for obtaining representations capturing wide contexts of words. On trained vector representations, there can be two types of dimensions (variables) Aida-bollegala-2024-semantic ; sense-aware dimensions that are involved with the sense changes and uniform dimensions that are not involved. Since conventional distance metrics (i.e. cosine or euclidean distances) are not able to distinguish these two types of dimensions, Aida-bollegala-2024-semantic addresses the importance of a distance metric that can accurately measure the semantic change. This paper proposes MMD-Sense-Analysis for realizing the two goals of identifying word sense changes and explaining the sense changes over periods, while distinguishing the sense-aware dimensions. The core idea of MMD-Sense-Analysis is selecting semantic-aware dimensions using a variable selection based on Maximum Mean Discrepancy (MMD), and qualifying similarities of two time periods. Maximum Mean Discrepancy (MMD) is a distance metric between two probability distributions and the popular choice for dataset shift adaptation or domain adaptation Yan-2017mind ; wang2021-rethinking . Since the word sense change can be regarded as a problem of detecting shift, MMD is a suitable distance metric for comparing two time periods. To my best knowledge, this is the first work using MMD for the task of word sense change detection. Figure 1 depicts the pairwise qualified similarities on multiple time periods. Table 1 represents selected words concerning the sense-aware dimension, and Figure 3 shows the selected words’ diachronic sense changes over multiple periods. This paper is organized as follows. Section 2.1 introduces the Maximum Mean Discrepancy (MMD) and the variable selection framework in optimizing an MMD estimator. Section 2.2 presents the sense change detection algorithm, MMD-Sense-Analysis . Section 3 describes experimental results. The codebase is available at the author’s Github repository 1 1 1 https://github.com/Kensuke-Mitsuzawa/mmd-tst-variable-selection-word-sense-change-analysis . 2 Proposed Method The proposed algorithm MMD-Sense-Analysis selects variables of word embeddings that are involved with the sense changes ( semantic aware dimensions Aida-bollegala-2024-semantic ), and then the proposed algorithm selects vocabularies (words) significantly influenced by the selected variables. In this paper, I call such variables sense change variables . I, firstly, formalize preliminaries. There is T\in\mathbb{N} time periods to be studied, and word embedding vectors E_{t}=\{e_{1},...,e_{V}\},e\in\mathbb{R}^{D} are given for a time period index t , where V is a set of vocabularies that the word embedding models. Essentially, MMD-Sense-Analysis can handle any word embedding models, however, it is recommended to use a word embedding model designed for treating the time-space, such as PMI-SVD joint Aida-2021 . MMD-Sense-Analysis selects sense change variables by the variable selection problem. The variable selection problem is defined as selecting a subset S\subset\{1,2,\dots,D\} of variables out of the D variables of word embedding vectors. The selected variables S represent indices of sense change variables . For the variable selection, I use the approach of “MMD Variable Selection” ( mitsuzawa-2023 ) . In the next section, I introduce Maximum Mean Discrepancy ( Gretton-2012a ) and the variable selection algorithm, and then, MMD-Sense-Analysis is introduced in Section 2.2 . 2.1 MMD Two-Sample Variable Selection Maximum Mean Discrepancy and MMD Variable Selection. Maximum Mean Discrepancy (MMD) is a kernel-based distance metric between probability distributions Gretton-2012a and a measure of distance between two probability distributions. It relies on a positive semi-definite kernel Hofmann-2008 ; Smola-2009 to map the distributions into a new space, a Reproducing Kernel Hilbert Space (RKHS), where probability distributions are represented as data points Muandet-2017 . A large MMD value indicates significant differences between two distributions. Given i.i.d. samples {\bf X}=\{{\bm{x}}_{1},\dots,{\bm{x}}_{n}\}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}P and {\bf Y}=\{{\bm{y}}_{1},\dots,{\bm{y}}_{n}\}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}Q , the MMD can be consistently estimated as \begin{split}&\widehat{\rm MMD}^{2}_{n}({\bf X},{\bf Y}):=\frac{1}{n(n-1)}\sum_{i\not=i^{\prime}}k({\bm{x}}_{i},{\bm{x}}_{i^{\prime}})+\frac{1}{n(n-1)}\sum_{j\not=j^{\prime}}k({\bm{y}}_{j},{\bm{y}}_{j^{\prime}})-\frac{2}{n^{2}}\sum_{i,j}k({\bm{x}}_{i},{\bm{y}}_{j}).\end{split} (1) Recently, two-sample variable algorithms based on MMD have been studied in the literature ( sutherland2017 ; wang2023variable ; mitsuzawa-2023 ) . Here, I explain the approach of ( mitsuzawa-2023 ) . First, I introduce Automatic Relevance Detection (ARD) kernel as the kernel k in Eq. ( 1 ), which is defined as k({\bm{x}},{\bm{y}})=\exp\left(-\frac{1}{D}\sum_{d=1}^{D}\frac{a_{d}^{2}(x_{d}-y_{d})^{2}}{\gamma^{2}_{d}}\right),\quad{\bm{x}}:=(x_{1},...,x_{D})^{\intercal}\in\mathbb{R}^{D},\ {\bm{y}}:=(y_{1},...,y_{D})^{\intercal}\in\mathbb{R}^{D} (2) where a_{1},...,a_{D}\geq 0 are called ARD weights , and \gamma_{1},\dots,\gamma_{D}>0 are constants to unit-normalize each variable. I compute \gamma_{1},\dots,\gamma_{D} by using the dimension-wise median (or mean) heuristic ; see ( mitsuzawa-2023 , Appendix C) for details. sutherland2017 optimizes the ARD weights a_{1},\dots,a_{D} in Eq.( 2 ) by maximizing the test power (i.e., the rejection probability of the null hypothesis P=Q when P\not=Q ) of a two-sample test based on the MMD estimate Eq.( 1 ). mitsuzawa-2023 proposes the variable selection by introducing L_{1} regularization Tibshirani-1996 to encourage sparsity for variable selection: \underset{a\in\mathbb{R}^{D}}{\min}-\log(\ell(a_{1},...,a_{D}))+\lambda\sum_{d=1}^{D}|a_{d}|,\quad\text{where}\quad\ell(a_{1},...,a_{D}):=\frac{\widehat{\rm MMD}^{2}_{n}({\bf X},{\bf Y})}{\sqrt{\mathbb{V}_{n}({\bf X},{\bf Y})+C}}. (3) In the above display, \lambda>0 is a regularization constant, C=10^{-8} is a constant for numerical stability, and \mathbb{V}_{n}({\bf X},{\bf Y}) is an empirical estimate of the variance of \widehat{\rm MMD}^{2}_{n}({\bf X},{\bf Y}) ; see ( mitsuzawa-2023 ) for the concrete form. The weights a_{1},\dots,a_{D} with larger \ell(a_{1},\dots,a_{D}) make the test power higher, thus making the two data sets {\bf X} and {\bf Y} more distinguishable. Minimizing the negative logarithm of \ell(a_{1},\dots,a_{D}) plus the L_{1} regularization term \lambda\sum_{d=1}^{D}|a_{d}| thus yield sparse weights a_{1},\dots,a_{D} that distinguish {\bf X} and {\bf Y} . Such weights can be used for selecting variables where the two data sets differ significantly. The regularization constant \lambda controls the sparsity of optimized ARD weights, and

Read the complete text