Semantic similarity estimation for domain specific data using BERT and other techniques
R. Prashanth · 2025
Read Semantic similarity estimation for domain specific data using BERT and other techniques by R. Prashanth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18602.txt.
Introduction
R Prashanth compared ways in a study on judging how alike two texts are in meaning, with a focus on narrow fields. Such judgments feed search, question answering, grouping of documents, and translation. The study tested three leading tools, including the BERT model, on two sets of question pairs.
One set came from a private business source and the other from a public Quora set. BERT did much better than the other tools. Fine tuning during its training, which lets it learn patterns from the data at hand, is offered as the reason. The author concludes that BERT is the best choice for field-specific data among the tools tested.
Catalog record
| CREATOR | R. Prashanth |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18602
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Semantic similarity estimation for domain specific data using BERT and other techniques Thanks: This is a preprint version of an article accepted for publication in the proceedings of Machine Learning and Data Mining 2019. Affiliation: Fidelity Business Services India Pvt. Ltd., R. Prashanth Abstract Estimation of semantic similarity is an important research problem both in natural language processing and the natural language understanding, and that has tremendous application on various downstream tasks such as question answering, semantic search, information retrieval, document clustering, word-sense disambiguation and machine translation. In this work, we carry out the estimation of semantic similarity using different state-of-the-art techniques including the USE (Universal Sentence Encoder), InferSent and the most recent BERT, or Bidirectional Encoder Representations from Transformers, models. We use two question pairs datasets for the analysis, one is a domain specific in-house dataset and the other is a public dataset which is the Quora’s question pairs dataset. We observe that the BERT model gave much superior performance as compared to the other methods. This should be because of the fine-tuning procedure that is involved in its training process, allowing it to learn patterns based on the training data that is used. This works demonstrates the applicability of BERT on domain specific datasets. We infer from the analysis that BERT is the best technique to use in the case of domain specific data. 1 Introduction Semantic similarity estimation of text is an important and open research problem in natural language processing as well as in natural language understanding [ 1 ] . Two texts are semantically similar if they mean the same thing [ 6 ] . And this semantic similarity estimation lays foundation to various downstream applications including information retrieval, document clustering, word-sense disambiguation, machine translation, text summarization, text generation, question answering (QA), short answer grading, automatic essay scoring, semantic search, dialog and conversational systems [ 6 ] . Realizing the potential of this area, the Semantic Textual Similarity (STS) shared task has been held annually since 2012, providing a venue for evaluation of state-of- the-art algorithms and models for semantic similarity estimation [ 7 , 8 , 10 , 9 , 11 ] . In the application area of question answering, generally a knowledge base (KB) is maintained containing queries and associated answers [ 1 ] . As the same query can be asked in many different ways, it becomes important to find the most semantically similar query from the KB to recommend an answer. For instance, the answers for ”To buy stocks what’s the minimum” and ”what is the minimum amount to buy a stock” is same. And question answering is one of the key research areas at Fidelity. There are many research articles covering textual semantic similarity estimation [ 4 , 3 , 2 , 5 ] . Word embeddings such as word2vec [ 12 ] , Glove [ 13 ] and FastText [ 22 ] have been used widely in NLP to compute vector representation for words capturing the semantics that can be used to compute the semantic similarity between words. Although word embeddings effectively capture the semantics for words, they are more prone to yield limited performance when used for sentence embedding via averaging of word vectors or via some other way [ 5 , 4 , 3 ] . Reasons for this decreased performance could be because, usually word embeddings do not consider the word order into account and the word embeddings are learnt in an unsupervised manner. Due to these shortcomings, recent research has focused on embedding of larger text, such as sentences, paragraphs or documents, straightaway, rather than looking at word embeddings. A lot of these research has lead to the development of various pre-trained models which includes Google’s Universal Sentence Encoder (USE) [ 4 ] and Facebook’s InferSent [ 3 ] . Generally, these pre-trained models have been trained on both supervised as well as unsupervised tasks in order to capture as much universal semantic information as possible. And these models have shown better performance on various downstream tasks (as well as on transfer learning) as compared to word embeddings. Along with research on sentence embedding, there is a big focus in the area of language model pretraining which has shown to give improved performance in various NLP tasks including predicting the relationship between sentences [ 20 , 21 ] . Very recently, Google had open sourced a new language representation model called BERT [ 2 ] which stands for Bidirectional Encoder Representations from Transformers, and it has shown to outperform other techniques in many NLP tasks such as text classification, textual entailment, question answering and language inference, without substantial task-specific architecture modifications, easily making it among the best in the state-of-the-art methods. In this paper, we use sentence embedding pre-trained models including Universal sentence encoder and InferSent, and language representation model of BERT for estimating the semantic similarity for pairs of text inputs. Along with this, we also use string matching method using the Ratcliff/Obershelp pattern-matching algorithm [ 26 ] which acts as the baseline for our analysis. For the experiments, we use an in-house dataset and a public dataset from Quora, called the Quora Question Pairs (QQP) [ 25 ] . 2 Materials 2.1 Dataset details Quora dataset: The Quora dataset consist of question pairs (that were asked on Quora by its users) along with a label indicating semantic equivalence. There are 404290 question pairs where 149263 (37%) were similar and the rest 255027 (63%) were not similar. This dataset was general in nature, capturing the questions asked by the Quora users that were not specific to any domain. The dataset was pre-processed and after pre-possessing, three observations were removed that belonged to the positive class. Fidelity data: The Fidelity data set consist of user query pairs along with the label for similarity. There are 3757 query pairs where 1703 (45%) are similar and the rest 2054 (55%) are not similar. This data is domain specific and it contained questions pertinent to the financial services domain. 3 Methods A high level overview of the analysis carried out in the paper is as shown in Fig. 1 . As mentioned above, we use the Quora dataset and an in-house dataset as well for the experiments. The purpose of using Quora dataset is for the sake of comparison with a public dataset. BERT has already been applied on the QQP dataset and have shown to beat state-of-the-art results. The datasets are given as inputs to different techniques, starting with string matching using the Ratcliff/Obershelp algorithm which effectively acts as a baseline model, pre-trained models including USE and InferSent for embedding text, and also the language representation model of BERT. The string matching method using the Ratcliff/Obershelp algorithm basically measures the similarity of sequences based on the number of matching characters. To account for the order of words in the sequence, the sequence is tokenized followed by sorting the tokens alphabetically, and then joining them back into a string. Before the string matching procedure, the sequences are lemmatized for the purpose of normalization. The pre-trained models (USE and InferSent) for text embedding generate embeddings for the sequences or the input question pairs and then appropriate metric is used