Shifting NER into High Gear: The Auto-AdvER Approach

Filippos Ventirozos, Ioanna Nteka*, Tania Nandy*, Jozef Baca*, Peter Appleby*, Matthew Shardlow · 2024

Read Shifting NER into High Gear: The Auto-AdvER Approach by Filippos Ventirozos, Ioanna Nteka*, Tania Nandy*, Jozef Baca*, Peter Appleby*, Matthew Shardlow (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05655.txt.

Introduction

Filippos Ventirozos and co-authors build a labeling scheme and dataset for pulling facts out of used-car ads, in a 2024 paper. Their schema, Auto-AdvER, tags three kinds of spans in ad text: the car's condition, its history, and commercial offers such as warranties or finance deals. The work was shaped with industry users in mind.

Two annotators agreed on 78 percent of labels, a solid mark for this kind of task. The team then raced several models: fine-tuned encoder models beat large language models overall, with the best encoder on top, though the big models were more precise when they did tag. The authors offer the dataset as a step toward finer ad analytics, pricing tools, and buyer insight.

Catalog record

CREATORFilippos Ventirozos, Ioanna Nteka*, Tania Nandy*, Jozef Baca*, Peter Appleby*, Matthew Shardlow
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-05655

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Shifting NER into High Gear: The Auto-AdvER Approach Filippos Ventirozos, Ioanna Nteka*, Tania Nandy*, Jozef Baca*, Peter Appleby*, Matthew Shardlow Abstract This paper presents a case study on the development of Auto-AdvER, a specialised named entity recognition schema and dataset for text in the car advertisement genre. Developed with industry needs in mind, Auto-AdvER is designed to enhance text mining analytics in this domain and contributes a linguistically unique NER dataset 1 1 1 The dataset will be made available upon publication. . We present a schema consisting of three labels: ‘Condition’, ‘Historic’ and ‘Sales Options’. We outline the guiding principles for annotation, describe the methodology for schema development, and show the results of an annotation study demonstrating inter-annotator agreement of 92% F1-Score. Furthermore, we compare the performance by using encoder-only models: BERT, DeBERTaV3 and decoder-only open and closed source Large Language Models (LLMs): Llama, Qwen, GPT-4 and Gemini. Our results show that the class of LLMs outperforms the smaller encoder-only models. However, the LLMs are costly and far from perfect for this task. We present this work as a stepping stone toward more fine-grained analysis and discuss Auto-AdvER’s potential impact on advertisement analytics and customer insights, including applications such as the analysis of market dynamics and data-driven predictive maintenance. Our schema, as well as our associated findings, are suitable for both private and public entities considering named entity recognition in the automotive domain, or other specialist domains. 1 Introduction The automotive industry relies heavily on accurate data regarding vehicles that are manufactured, sold and resold. In particular digital businesses rely on information extracted from online sources, such as car advertisements, for various applications including market analysis, competitive intelligence and customer insights. Named entity recognition (NER) serves as a pivotal technique to transform unstructured text into structured data Li et al. 2020 . However, the unique nature of car advertisements necessitates a specialised NER label schema. This paper introduces a novel automobile advertisements NER schema developed within an industry context, named Auto-AdvER ( Auto mobile Adv ertisment E ntity R ecognition). In the present study, we introduce the guiding principles for forming the annotation labels and labelling their respective spans from text. This work introduces a novel framework for labelling car advertisements with key information. We anticipate the development of future schema which extend ours in subsequent research. With this schema, we hope to set a standard for text annotation studies within the automotive industry and beyond, demonstrating best practices for future development. By sharing our schema and data we also hope to encourage collaborative annotations from other interested parties from academia and industry. Specifically, our contributions lie in demonstrating: 1. A novel domain NER dataset 2. The comparison of several applied NER models 3. Critical discussion of these results, and applicability to the automotive industry and beyond. The paper is structured as follows: Section 2 introduces related NER studies in the automotive domain. In Section 3 , we describe our annotation guidelines and methodology, and in Section 3.5 we present the data properties and language. Section 4 presents our experiments and inter-annotator agreement (IAA) results. We discuss wider applicability and future work in Section 5 , and conclude in Section 6 . 2 Related Work NER is a foundational sequence labelling task in NLP, with early datasets developed for the CoNLL evaluation campaigns Tjong Kim Sang and De Meulder 2003 focusing on general news text. It involves identifying spans in text applicable for labelling and classifying them into predefined categories — a process often performed end-to-end ( Jehangir et al. 2023 ) . Typical NER approaches have employed sequence labelling models such as conditional random fields ( Keraghel et al. 2024 ) , BiLSTMs Chiu and Nichols 2016 or transformers Hanh et al. 2021 . NER has been explored in various domains. Its applicability is useful for different domains and industries and acts typically as the first step towards information extraction projects. NER has been applied from biomedical applications ( Fu et al. 2023 ) , agriculture ( G. et al. 2023 ) to music ( Epure and Hennequin 2023 ) . In these cases it is important to develop a bespoke annotation schema for the domain to represent the entities therein. To the best of our knowledge, the only study addressing NER in the automotive domain is that of Hu and Ma 2024 , who conducted NER on car accessories in Chinese. In contrast, our study investigates a more general set of named entities in car advertisements and is conducted in English. 3 Dataset Overview 3.1 Annotation Methodology To the best of our knowledge, this is the first schema for labelling car text advertisements. Consequently, we devised our own labels through the use of the action research framework ( Molineux 2018 ) . This work represents an effective collaboration between industry and academic partners, embedding NLP research into practice. Through our collaboration, we engaged extensively with industry data and resources and consulted with industry infrastructure engineers to leverage an in-depth understanding of the automotive domain. This in-house expertise was pivotal in crafting the initial set of labels and guidelines and to the success of our project, leading to the following core principles as outlined below. 3.1.1 Core Principles 1. To extract the named entities schema, we initially employed coarser NER labels, with the intention of refining these entities through a subsequent step of entity linking. 2. Each label was designed to be easily identifiable, typically answerable with a single question. 3. The aim was to annotate only factual statements, not opinions or personal sentiments. 3.1.2 Iterations Our methodology for crafting the guidelines was inspired by the DevOps approach, as outlined in Jabbari et al. 2016 . In DevOps two teams (Dev and Ops) collaborate through iterations of design and development compromising appropriately on design decisions to maximise efficiency. Initially the university-based dev elopment team created the classes and annotation guidelines. These were then discussed with the industry-based op eration s team comprised of domain-expert data scientists. The team had two members from the university partner (Dev) and three from the industry partner (Ops). Initial Iterations During the first set of iterations, the development team proposed a draft of the guidelines. These proposals were then refined by the operations team, focusing on two main aspects: 1. The inclusivity of each class and the extent to which they covered various aspects of the advertisements. 2. The identification of any omitted classes, which could either be sourced from other data or deemed non-essential. Following these discussions, the development team further refined the classes. When a class lacked clarity regarding its encompassed aspects, the team evaluated the semantic coherence of each aspect within the proposed definitions. Rapid prototyping enabled them to explore alternative classes and assess their detectability. Secondary Iterations As the labels began to take shape, we entered the second set of iterations. Here, the development team introduced annotation tasks for the operations team. After the operations team completed these tasks, their feedback was discussed to further refine the guidelines. This feedback included analysis of challenging cases and observations on whether additional aspects should be incorporated. 3.2 Named Entity Tags As a result of this process, we developed three named entity

Read the complete text