Positional Attention for Efficient BERT-Based Named Entity Recognition

Mo Sun, Siheng Xiong, Yuankai Cai, Bowen Zuo · 2025

Read Positional Attention for Efficient BERT-Based Named Entity Recognition by Mo Sun, Siheng Xiong, Yuankai Cai, Bowen Zuo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01868.txt.

Introduction

Mo Sun, Siheng Xiong, Yuankai Cai and Bowen Zuo present a cheaper way to fine-tune BERT for named entity recognition in 2025. Retraining the full model for each new task is slow and costly, so they add positional attention to the recognition process and reuse pre-trained parameters.

The framework was tested on a Kaggle dataset drawn from the Groningen Meaning Bank corpus. It reached strong accuracy with fewer training epochs than full fine-tuning. The authors offer the method as a practical route for teams that need custom entity recognition, for example in documents or support tickets, without paying for long training runs each time.

Catalog record

CREATORMo Sun, Siheng Xiong, Yuankai Cai, Bowen Zuo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01868

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Positional Attention for Efficient BERT-Based Named Entity Recognition Mo Sun, Siheng Xiong, Yuankai Cai, Bowen Zuo Abstract This paper presents a framework for Named Entity Recognition (NER) leveraging the Bidirectional Encoder Representations from Transformers (BERT) model in natural language processing (NLP). NER is a fundamental task in NLP with broad applicability across downstream applications. While BERT has established itself as a state-of-the-art model for entity recognition, fine-tuning it from scratch for each new application is computationally expensive and time-consuming. To address this, we propose a cost-efficient approach that integrates positional attention mechanisms into the entity recognition process and enables effective customization using pre-trained parameters. The framework is evaluated on a Kaggle dataset derived from the Groningen Meaning Bank corpus and achieves strong performance with fewer training epochs. This work contributes to the field by offering a practical solution for reducing the training cost of BERT-based NER systems while maintaining high accuracy. 1 Introduction This project uses a language representation model called Bidirectional Encoder Representations from Transformers (BERT) to solve a common problem in natural language processing (NLP). The BERT model can be used in NLP for classification, predicting whether a sentence is grammatically correct or not, and many other applications. Unlike recent language sensation models brown1992class , BERT pre-trains deep representations from unlabeled text using the context of the text around it devlin2018bert . The pre-trained BERT model can then be fine-tuned to suit specific applications by using another dense layer. The task of named entity recognition (NER) is to recognize the named entities in the given text. For example, it can classify words as names, verbs, etc. NER is a preliminary and important task in NLP and can be used in many downstream NLP tasks, such as relation extractions bunescu2005shortest , event extraction chen2015event , and question answering yao2014information . In recent years, numerous methods have been studied for NER tasks, including Hidden Markov Models (HMMs) bikel1997nymble , Support Vector Machines (SVMs) isozaki2002efficient , and Conditional Random Fields (CRFs) lafferty2001conditional . The BERT model is a state-of-the-art model in NLP to do entity recognition in a Google project, and the training set of this model is the Wikipedia dataset. So, when training the model, Google used huge storage for the whole dataset and Tpu clusters to handle this task in a few days. However, if we use it in an application, we need to customize this model to our dataset. It is easy to collapse and hard to get ideal results with the pre-trained parameters. We decided to build a framework that doesn’t need to train the BERT model from scratch (using pre-trained parameters) and can get good results by using fewer training epochs. One challenge of doing this is that the BERT model is large and expensive to train and fine-tune, so one must be careful to choose methods that have low enough time and space costs. This project is important because to apply natural language models in different situations, a lot of companies ask the Google cloud solution team to help them train a model to fit their needs. Although Google can train models based on customers’ needs, it is also a high cost and slow processing to build the framework for each company. So, we want to find a solution to reduce the cost of training the process of the BERT model. The research gap we are filling is to introduce positional attention mechanisms in entity recognition and try to find a low-cost method to customize models with pre-trained parameters. Therefore, our project will demonstrate a way that the BERT model can easily be customized to a particular need or context efficiently. This project used a dataset on Kaggle which is extracted from the Groningen Meaning Bank corpus that is tagged, annotated, and built specifically to train the classifier to predict named entities such as name, location, etc. This dataset contains a huge amount of combinations of words and is updated regularly, which is helpful for us to train the model. 2 Methodology In our study, we utilize the BERT model, which is based on the state-of-the-art methods described in the papers “Attention is All you Need” and “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.” The transformer architecture relies on attention mechanisms, eliminating the need for recurrence. Attention mechanisms help determine the focus or emphasis placed on each word in a sentence while considering each individual word. The pre-trained BERT model converts each word in a sentence into Word Piece embeddings and processes them through 12 layers of encoders, each containing a multi-headed self-attention layer and a feed-forward neural network, as shown in figure 1. Figure 1 : BERT’s 12 layers of two-part encoders The Word Piece embeddings are how each word is represented in BERT devlin2018bert . Each word is a vector of 768 features that describes the meaning of the word. Words with similar meanings will have embeddings that are close to each other, such as “couch” and “sofa,” whereas words with different meanings, such as “tree” and “spaceship” will have embeddings that are numerically far apart from each other. BERT accounts for the relative positions of word encodings, enabling it to differentiate between sentences like ”Dogs chase cats” and ”Cats chase dogs.” BERT is pre-trained on various tasks, such as masked language modeling, where some words are replaced by a [MASK] token, and the model has to predict the original word. In this project, we build upon the pre-trained BERT model to address Named Entity Recognition (NER) for a specific text dataset from Kaggle. We leverage the pre-trained BERT model’s knowledge while fine-tuning it for our domain-specific task. Our approach involves the following steps: a. Preprocessing the text dataset: Tokenizing the sentences, converting words into Word Piece embeddings, and encoding their positions. b. Fine-tuning the pre-trained BERT model: Adapting the model to our specific NER task by training it on our domain-specific dataset, which allows the model to learn the relevant vocabulary and context. c. Evaluating the model: Assessing the performance of the fine-tuned BERT model on our NER task using standard evaluation metrics, such as precision, recall, and F1-score. Comparing the performance of our approach with other existing methods to understand its effectiveness in the given context. We believed that using the BERT model for solving the NER problem would be successful for several reasons: 1. Proven success in NLP tasks: The BERT model has shown great performance in various NLP tasks, outperforming many previous models. Its pre-training on large text corpus and attention mechanism has proven effective in understanding context and relationships between words. 2. Transfer learning: The pre-trained BERT model provides a solid foundation that can be fine-tuned for domain-specific tasks. 3. Adaptability to context: Fine-tuning BERT for our specific NER task allows it to learn the relevant domain-specific vocabulary and context, making it more suitable for identifying named entities in our dataset. The creative side of our approach mostly comes from these two parts below. 1. Customized fine-tuning: Although BERT has been used for NER tasks before, our approach fine-tunes the model specifically for our domain and dataset, potentially leading to better performance in our problem domain. 2. Exploration of different techniques: Our approach could

Read the complete text