Extracting domain-specific terms using contextual word embeddings Journal: Expert systems with applications
Andraz Repar, Nada Lavrac, Senja Pollak · 2025
Read Extracting domain-specific terms using contextual word embeddings Journal: Expert systems with applications by Andraz Repar, Nada Lavrac, Senja Pollak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17278.txt.
Introduction
Andraž Repar, Nada Lavrač, and Senja Pollak improved term extraction in two thousand twenty-five for specialist Slovenian texts. Picking the true terms of a field out of its papers is a core step for glossaries and search. Their method joins classic features from older extractors with fresh context features from contextual word embeddings.
Working from a newly annotated Slovenian corpus, they derived rules for candidate terms, then trained a support vector machine on statistical, linguistic, and context features. Across four fields, from biomechanics to veterinary science, their scores beat the prior best for Slovenian by a clear margin. Context embeddings, they conclude, are a real asset for this task.
Catalog record
| CREATOR | Andraz Repar, Nada Lavrac, Senja Pollak |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17278
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Extracting domain-specific terms using contextual word embeddings Journal: Expert systems with applications Andraz Repar, Nada Lavrac, Senja Pollak Abstract Automated terminology extraction refers to the task of extracting meaningful terms from domain-specific texts. This paper proposes a novel machine learning approach to terminology extraction, which combines features from traditional term extraction systems with novel contextual features derived from contextual word embeddings. Instead of using a predefined list of part-of-speech patterns, we first analyse a new term-annotated corpus RSDO5 for the Slovenian language and devise a set of rules for term candidate selection and then generate statistical, linguistic and context-based features. We use a support-vector machine algorithm to train a classification model, evaluate it on the four domains (biomechanics, linguistics, chemistry, veterinary) of the RSDO5 corpus and compare the results with state-of-art term extraction approaches for the Slovenian language. Our approach provides significant improvements in terms of F1 score over the previous state-of-the-art, which proves that contextual word embeddings are valuable for improving term extraction. 1 Introduction Automated terminology extraction (ATE) refers to the task of extracting meaningful terms from domain-specific texts. Terms are single-word (SWU) or multi-word units (MWU) of knowledge, which are relevant for a particular domain. Since manual identification of terms is costly and time consuming, ATE approaches can reduce the effort needed to generate relevant domain-specific terms. Recognizing and extracting domain-specific terms, which is useful in various fields, such as translation, dictionary creation, ontology generation and others, remains a difficult task. ATE has been the subject of research for a long time, but state-of-the-art performance often lags behind other NLP tasks. The most important reason for this is the fact that it is very difficult to provide a clear definition of the nature of domain-specific terms and, as a result, annotated datasets for ATE are relatively scarce. Nevertheless, in the recent years, ATE-specific ACTER datasets for English, French and Dutch were released and used in several papers describing state-of-the-art ATE approaches ( terryn2020no ) , while the recently released RSDO5 dataset for Slovenian ( Jemec Tomažin et al. 2021 ) is used in the present paper. In this paper, we describe a terminology extraction methodology that combines two traditional aspects of ATE with a novel contextual-embedding approach in a machine learning setting. Focusing on the Slovenian language, which is an under-resourced Slavic language with a rich morphology, we conducted the first experiments on a new RSDO5 corpus of term-annotated texts created as part of the RSDO national project. The proposed approach starts with the corpus — we first analyze the annotated terms in the corpus and study their part-of-speech tags. While traditional systems use a set of pre-defined part-of-speech patterns to identify the initial candidate terms (CTs), we take a different approach and instead define a shallow filter for CTs, which considers only very basic part-of-speech based information. In our approach, we generate three types of features (linguistic, statistical and contextual) and use them to train a linear support vector machine (SVM) classifier. Since the RSDO5 corpus contains four different domains, we train the algorithm on three domains and test its performance on the fourth domain using the standard measures of precision, recall and the F1 score (as in the related work by rigouts2020termeval ). The rest of this paper is organized as follows: Section 2 presents the related work, Section 3 introduces the term-annotated corpus used in the experiments, Section 4 contains a description of the term extraction system, Section 5 contains the results of the experiments, and Section 6 presents the conclusions and ideas for future work. 2 Related work ATE systems were traditionally classified as either statistical, linguistic or a combination of these two approaches. The linguistic approach utilizes the distinctive linguistic aspects of terms, most often their syntactic (i.e. part-of-speech) patterns. On the other hand, the statistical approach takes advantage of term frequencies in a corpus. However, most traditional systems are hybrid, using a combination of the two approaches. For example, Justeson and Katz 1995 first define part-of-speech (POS) patterns of terms and then use simple frequencies to filter the term candidates. Many terminology extraction algorithms are based on the concepts of termhood and unithood defined by Kageura and Umino 1996 : termhood is “the degree to which a stable lexical unit is related to some domain-specific concepts” and unithood is “the degree of strength or stability of syntagmatic combinations and collocations”. Termhood-based statistical measures ( vintar2010bilingual ) function on a presumption that a term’s relative frequency will be higher in domain-specific corpora than in the general language, while common statistical measures, such as mutual information ( Daille et al. 1994 ) , are used to measure unithood. These two approaches have been used as a basis of several hybrid systems, such as Termolator ( Meyers et al. 2018 ) and TermEnsembler ( repar2019termensembler ) . However, in the last decade, this division has become too simplistic due to the emergence of new machine learning and deep learning approaches that could not be classified as either linguistic, statistical or hybrid in the traditional sense. The advances in embeddings and deep neural networks have also influenced the terminology extraction field. Amjadian et al. 2016 were one of the first to leverage embeddings for terminology extraction by trying to represent unigram terms as a combination of local and global vectors. Other works involving non-contextual word embeddings include the approaches that either devise a co-training system involving two neural networks to determine whether a term is domain relevant or not ( wang2016featureless ) , use word embeddings to estimate term similarity in a graph-based ranking system ( Khan et al. 2016 ) , employ word embeddings to measure semantic relatedness of term candidates in order to re-rank term candidates generated with traditional term extraction methods ( zhang2017semre ) , identify term candidates using sequence labeling and word-level and character-level embeddings ( Kucza et al. 2018 ) , or devise a nested term extraction classifier with features from various (non-contextual and contextual) word embedding models ( Gao and Yuan 2019 ) . Contextual word representations, such as eLMO ( peters-etal-2018-deep ) and BERT ( Devlin et al. 2019 ) , can encode additional information about terms as illustrated by the fact that the winning approach in the TermEval2020 competition ( rigouts2020termeval ) uses the BERT model. TALN-LS2N ( Hazem et al. 2020 ) , the winning approach for English and French, uses BERT in a binary classification setting, where a combination of n-grams and a sentence are used as an instance and the classifier needs to determine for each n-gram inside the sentence whether it is a term or not. On the other hand, the winning approach for Dutch ( rigouts2020termeval ) uses pretrained GloVe word embeddings that are fed into a bi-directional LSTM-based neural architecture. Another well-performing system used in this competition that combines word embeddings with statistical approaches compares the performance of terminology extraction built on an improved TextRank, TFIDF, clustering, and termhood features ( Păi˙ 2020 ) . Recently, state-of-the-art results for English, French and Dutch have been reported by Lang et al. 2021 using a sequence labeling approach. Another distinct approach is to utilize machine learning with feature engineering. It involves first extracting