ViConBERT: Context-Gloss Aligned Vietnamese Word Embedding for Polysemous and Sense-Aware Representations
Khang T. Huynh, Dung H. Nguyen, Binh T. Nguyen 🖂 · 2025
Read ViConBERT: Context-Gloss Aligned Vietnamese Word Embedding for Polysemous and Sense-Aware Representations by Khang T. Huynh, Dung H. Nguyen, Binh T. Nguyen 🖂 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12249.txt.
Introduction
Khang T. Huynh, Dung H. Nguyen and Binh T. Nguyen present ViConBERT, a model for Vietnamese word meanings in context. Most progress on sense-aware embeddings has come in English, while Vietnamese lacks strong models and test resources. Words with several senses are a hard case, since the right meaning depends on the sentence around them.
ViConBERT learns embeddings with contrastive training and with distillation from dictionary glosses, so senses are tied to definitions. The team also builds ViConWSD, a large synthetic dataset for word sense and similarity tests in Vietnamese. Results beat strong baselines on sense disambiguation and stay competitive on graded similarity measures.
Catalog record
| CREATOR | Khang T. Huynh, Dung H. Nguyen, Binh T. Nguyen 🖂 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-12249
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ViConBERT: Context-Gloss Aligned Vietnamese Word Embedding for Polysemous and Sense-Aware Representations Khang T. Huynh, Dung H. Nguyen, Binh T. Nguyen 🖂 Abstract Recent advances in contextualized word embeddings have greatly improved semantic tasks such as Word Sense Disambiguation (WSD) and contextual similarity, but most progress has been limited to high-resource languages like English. Vietnamese, in contrast, still lacks robust models and evaluation resources for fine-grained semantic understanding. In this paper, we present ViConBERT , a novel framework for learning Vietnamese contextualized embeddings that integrates contrastive learning (SimCLR) and gloss-based distillation to better capture word meaning. We also introduce ViConWSD , the first large-scale synthetic dataset for evaluating semantic understanding in Vietnamese, covering both WSD and contextual similarity. Experimental results show that ViConBERT outperforms strong baselines on WSD (F1 = 0.87) and achieves competitive performance on ViCon (AP = 0.88) and ViSim-400 (Spearman’s ρ \rho = 0.60), demonstrating its effectiveness in modeling both discrete senses and graded semantic relations. Our code, models, and data are available at https://github.com/tkhangg0910/ViConBERT . 1 Introduction Understanding word meaning in context is central to tasks such as word sense disambiguation (WSD) and contextual semantic similarity. It remains a fundamental challenge in NLP with broad applications in machine translation, question answering, and information retrieval. Accurate contextual modeling enables systems to distinguish between multiple senses of a word and capture subtle semantic nuances crucial for downstream tasks. While substantial progress has been made in high-resource languages like English with transformer-based models [ 7 ] , advances for low-resource languages such as Vietnamese remain limited. Fine-grained semantic understanding in Vietnamese faces two main gaps: (i) the absence of embedding models explicitly trained for sense-level discrimination, and (ii) the scarcity of publicly available benchmark datasets for contextual word semantic evaluation. Recent approaches integrating contextual embeddings with contrastive learning show promise for capturing semantic distinctions but are mostly developed for high-resource settings. For Vietnamese, available datasets are either small, inaccessible [ 15 , 18 ] , or not tailored for WSD and contextual word similarity [ 6 , stsbvn ] , leaving the field without a standardized basis for progress. To bridge this gap, we introduce a novel framework for learning contextualized Vietnamese word embeddings via word–gloss distillation. In addition, we construct ViConWSD, a new synthetic benchmark dataset tailored for evaluating fine-grained semantic understanding in Vietnamese. By systematically generating sense-labeled examples from gloss definitions, our dataset provides large-scale supervision where manual annotations are infeasible. Our key contributions are: 1. We propose a training pipeline that leverages gloss alignment to enhance contextual embeddings in low-resource settings, with a focus on Vietnamese. 2. We curate and release ViConWSD, the first large-scale synthetic benchmark for Vietnamese WSD and contextual similarity. 3. We conduct extensive experiments showing consistent improvements over strong baselines across multiple semantic tasks. 2 Related Works Transformer-based models like BERT and RoBERTa [ 7 , 11 ] achieve strong results on semantic tasks via context-sensitive embeddings. Multilingual variants (XLM-R [ 5 ] ) and Vietnamese-specific models (PhoBERT, ViDeBERTa [ 14 , 17 ] ) extend these advances to Vietnamese. Yet their generic objectives capture broad context but struggle with fine-grained semantics. To address such challenges, WSD and semantic similarity have been extensively studied. Early WSD approaches relied on lexical resources like WordNet [ 13 ] , with evaluation datasets such as SemCor and SemEval [ 1 ] . Recent neural methods [ 10 ] integrate gloss definitions (PolyBERT, BEM [ 19 , 2 ] ), with contrastive learning increasingly used to align contextual and gloss embeddings [ 4 ] . Similarly, semantic similarity has progressed from static embeddings [ 12 , 3 ] to contextualized approaches like SimCSE [ 8 ] . Standard benchmarks include SimLex-999 [ 9 ] and WiC [ 16 ] . Despite this progress, most methods and resources focus on high-resource languages. Vietnamese NLP still faces a scarcity of large-scale semantic datasets: resources like Vietnamese WordNet support syntax and tagging, while semantic datasets such as ViCon and VSimLex-999 [ 15 , 18 ] remain limited in size and coverage. This motivates our work to introduce a scalable semantic framework for Vietnamese, combining contrastive learning with gloss embeddings and providing richer evaluation resources for fine-grained semantic understanding. 3 Our Proposed ViConBERT Model Figure 1 : Training architecture of ViConBERT. Left (red): the Context Encoder processes sentences (e.g., ”Anh ấy đang khoan tường.” He is drilling the wall) with the target word khoan (drill) via multi-head attention and projection to produce contextual embeddings. Right (blue): the Gloss Encoder encodes glosses (e.g., ”Hành động tạo ra một lỗ…” An action that creates a hole…) into gloss embeddings. The objective combines InfoNCE to align context and gloss and Semantic Structure Loss to preserve their relative semantic structure. Context block is trainable; gloss block is frozen. We propose ViConBERT (Vietnamese-Context BERT), a model designed to represent the semantic space of contextualized word embeddings. The overall architecture is illustrated in Figure 1 . The key idea behind ViConBERT is to distill the gloss embedding space of a well-pretrained sentence embedding model into a context encoder through a contrastive learning framework [ 4 ] . Specifically, the model aligns the contextual representation of a target word with the vector representation of its corresponding gloss (definition). Unlike previous gloss-based architectures such as GlossBERT , BEM , and PolyBERT that treat word sense disambiguation as a discrete classification task, ViConBERT models sense semantics in a continuous embedding space . This allows the model to capture graded similarity between related senses rather than assigning isolated sense labels. Moreover, by distilling the gloss space from a pretrained sentence embedding model, ViConBERT obtains stable semantic anchors for each word, making the training more efficient and less dependent on complex negative sampling strategies. Let C be a tokenized context sentence containing the target word W at token indices I . The context block B_{c} produces a contextual embedding E_{C}=B_{c}(C,I) . Similarly, let G denote the gloss corresponding to W . The gloss block B_{g} generates the gloss embedding E_{G}=B_{G}(G) . We use C_{B} and G_{B} to denote the matrices containing all E_{C} and E_{G} vectors in a mini-batch. 3.1 Context Block And Gloss Block Context Block. The context encoder f_{c} is a pretrained BERT-based model that processes input tokens C=[T^{1},\dots,T^{n}] to produce contextual embeddings H_{C}=f_{c}(C)\in\mathbb{R}^{n\times d_{h}} . We extract embeddings at the target word span I=(start\_pos,end\_pos) and apply mean pooling to obtain query vector Q\in\mathbb{R}^{1\times d_{h}} . Using multi-head attention with Q as query and H_{C} as key-value, we compute: \tilde{H_{c}}=\text{MultiHeadAttn}(Q,H_{C},H_{C}) (1) Finally, we project to the joint embedding space: E_{C}=\tilde{H_{c}}\cdot W_{C} (2) where W_{C}\in\mathbb{R}^{d_{h}\times d_{model}} . Gloss Block. We employ a pretrained sentence embedding model optimized for semantic textual similarity as the gloss encoder f_{g} : E_{G}=f_{g}(G) (3) 3.2 Objective Function Gloss-Context Alignment Loss ( \mathcal{L}_{\text{InfoNCE}} ) encourages context embeddings to align with corresponding gloss embeddings using InfoNCE loss: \mathcal{L}_{\text{InfoNCE}}=\frac{1}{N}\sum_{i=1}^{N}-\log\left(\frac{\sum_{j\in P_{i}}\exp(g_{i}^{\top}c_{j}/\tau)}{\sum_{k\neq i}\exp(g_{i}^{\top}c_{k}/\tau)+\epsilon}\right) (4) where \tau is temperature and P_{i} denotes positive pairs for sample i . Semantic Structure Loss ( \mathcal{L}_{\text{SS}} ) is our proposed loss that promotes structural consistency between the context and gloss embedding spaces. Specifically, it encourages the relative distances (i.e., semantic topology) among context embeddings to mirror those among their corresponding gloss embeddings. We compute dissimilarity