Is ’Hope’ a Person or an Idea? A Pilot Benchmark for NER: Comparing Traditional NLP Tools and Large Language Models on Ambiguous Entities
Payam Latifi · 2025
Read Is ’Hope’ a Person or an Idea? A Pilot Benchmark for NER: Comparing Traditional NLP Tools and Large Language Models on Ambiguous Entities by Payam Latifi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12098.txt.
Introduction
A tiny, hand-labeled benchmark puts a sharp question to named entity recognition: when a word like Hope could be a person or an idea, who reads the context better? Payam Latifi runs the pilot comparison solo in 2025 across six systems.
Three classic tools, NLTK, spaCy, and Stanza, face three general-purpose language models, Gemini, DeepSeek, and Qwen, on a dataset of about a hundred nineteen tokens spanning persons, locations, organizations, dates, and times. The language models generally win on context-sensitive names, with Gemini posting the best average score, while Stanza stays more consistent on structured tags like locations and dates. The models wobble on time expressions and multi-word organizations, leaving practical advice on which system suits which job.
Catalog record
| CREATOR | Payam Latifi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-12098
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Is ’Hope’ a Person or an Idea? A Pilot Benchmark for NER: Comparing Traditional NLP Tools and Large Language Models on Ambiguous Entities Payam Latifi Abstract This pilot study presents a small-scale but carefully annotated benchmark of Named Entity Recognition (NER) performance across six systems: three non-LLM NLP tools (NLTK, spaCy, Stanza) and three general-purpose large language models (LLMs: Gemini-1.5-flash, DeepSeek-V3, Qwen-3-4B). The dataset contains 119 tokens covering five entity types (PERSON, LOCATION, ORGANIZATION, DATE, TIME). We evaluated each system’s output against the manually annotated gold standard dataset using F1-score. The results show that LLMs generally outperform conventional tools in recognizing context-sensitive entities like person names, with Gemini achieving the highest average F1-score. However, traditional systems like Stanza demonstrate greater consistency in structured tags such as LOCATION and DATE. We also observed variability among LLMs, particularly in handling temporal expressions and multi-word organizations. Our findings highlight that while LLMs offer improved contextual understanding, traditional tools remain competitive in specific tasks, informing model selection. K eywords Named Entity Recognition, Large Language Models, NLP Benchmarks, Entity Ambiguity, Performance Evaluation 1 Introduction Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), aiming to identify and classify named entities such as persons, locations, organizations, dates, and time expressions within unstructured text. NER plays a crucial role in downstream applications like information extraction, question answering, and semantic search, making it an essential building block for modern language technologies [ 1 ] . In recent years, NER systems have evolved significantly, ranging from rule-based and statistical models to deep learning and transformer-based architectures [ 2 ] . Comprehensive reviews of earlier techniques appear in [ 3 ] . Recent surveys highlight LLM potential for few-shot NER [ 4 ] ; the present study employs single-shot prompting for each LLM. Non-LLM NLP tools such as NLTK, spaCy, and Stanza provide efficient, deterministic solutions that are widely used in practical applications due to their speed and reliability. However, the emergence of large language models (LLMs) like Gemini, DeepSeek, and Qwen has introduced new possibilities for more context-aware and flexible entity recognition. This paper addresses: What performance patterns emerge when comparing traditional NLP libraries and LLMs on a small, ambiguity-rich dataset, and how do these inform task-specific model selection? This paper presents an exploratory, small-scale comparative analysis of non-LLM NER tools and LLMs, focusing on their performance across five key entity types: PERSON, LOCATION, ORGANIZATION, DATE, and TIME. We manually annotated a small but representative dataset and evaluated each tool’s ability to align with human annotation using standard evaluation metrics such as accuracy, precision, recall, and F1-score. Our methodology includes: Manual annotation of a custom dataset, Token-level alignment of model outputs, Quantitative evaluation of six NER systems, Visual and statistical comparison of performance across models and entity types. Given the deliberately small scale of our dataset (119 tokens), the study is framed as an exploratory pilot, prioritizing qualitative insights and reproducibility over large-scale generalization. This work’s novelty lies in its focus on ambiguity (e.g., ‘Hope’ as PERSON) in a small dataset, enabling detailed error analysis unattainable in large corpora. Because the sample contains only 119 tokens (56 entities), numerical differences should be interpreted as directional only; we report no statistical significance tests. 2 Data 2.1 Source and Collection The dataset used in this study consists of manually selected and annotated sentences focusing on named entity recognition (NER) tasks. The input text was compiled from various fictional and descriptive narratives, including event descriptions and character interactions. While not sourced from a public corpus or digital archive, the dataset was constructed to reflect real-world NER tagging challenges such as ambiguity, multi-word entities, and overlapping entity types. The dataset was tokenized with NLTK’s Punkt word_tokenize , resulting in 119 tokens, which served as the reference tokenization for all systems. All model outputs were manually aligned to this token list to ensure consistent evaluation across tools. The full input text used for evaluation is shown below: On a chilly Thursday morning, April Blake met Justice Hope at Madison Square, near Washington Tower. At 8:45, a memo from the Center for Civic Leadership reached the office of River Clark, head of Northern Union Trust. Later, Horizon filed a complaint dated October 2018 with the Department of Historical Records. By midday, journalists from The Chronicle and Liberty Press were circling City Hall. Jordan Reed mentioned a meeting on the fourth Friday of March. Meanwhile, Trinity Wells, formerly of Bridgewater School, was seen near Lincoln. Just before dusk, the Mayor’s Office received a tip linked to Ashley Fields and Pine. 2.2 Annotation Process To ensure a reliable gold standard for comparison, all tokens were manually annotated by the author based on established NER conventions. The annotation followed these principles: PERSON Names of individuals (e.g., April, Jordan Reed) LOCATION Geographic locations or place names (e.g., Madison Square, Washington Tower, Lincoln, City Hall) ORGANIZATION Organizations, institutions, or companies (e.g., Center for Civic Leadership, Northern Union Trust) DATE Specific calendar dates or named time periods (e.g., October 2018, the fourth Friday of March) TIME Expressions referring to specific times of day (e.g., 8:45, dusk) Each token was labeled individually, ensuring alignment with tokenization standards used by NLP tools (see Appendix C). 2.3 Dataset Characteristics Although compact, the dataset is engineered to maximize diagnostic value: it packs 119 tokens with exactly the kinds of surface-level traps that historically expose gaps between rule-based, neural and prompt-based taggers. Ambiguous person names such as “Justice” and “Hope” force systems to decide whether a capitalized common noun denotes a role or an individual, a distinction on which lexicon-driven models often flip. Multi-token spans (“Madison Square”, “Northern Union Trust”, “the fourth Friday of March”) test whether taggers correctly open and close brackets across stop-words and prepositional phrases; single-token oversights here cascade into four or five separate errors under token-level evaluation. Finally, context-sensitive temporal expressions (“midday”, “dusk”) require pragmatic, document-level inference rather than gazetteer lookup, penalizing models that rely solely on explicit date strings such as “October 2018”. By concentrating these phenomena in one short passage we can observe boundary-decision and sense-disambiguation strategies side-by-side without the noise of a larger corpus. 2.4 Entity Distribution Table 1: Distribution of Entity Types in the Annotated Dataset (n = 119 tokens) Entity Type Count Percentage DATE 7 5.88% LOCATION 8 6.72% ORGANIZATION 22 18.49% PERSON 13 10.92% TIME 6 5.04% O (non-entities) 63 52.94% Total Tokens 119 100% The observed distribution of entities reflects typical patterns found in real-world based corpora, where organizational entities are abundant while temporal expressions are relatively scarce. Across the 119 tokens, non-entity tokens (O) constitute 63 instances (52.9%), while the remaining 56 tokens are classified as entities. Among these, ORGANIZATION is the most frequent category, accounting for 22 tokens (18.5%), largely due to multi-word institutional names such as “Center for Civic Leadership” and “Northern Union Trust”. PERSON entities appear 13 times (10.9%), including both unambiguous names like “April Blake” and ambiguous cases such as “Justice Hope”. LOCATION comprises eight tokens (6.7%), covering multi-word place names like “Madison Square” and “Washington Tower”, as well as single toponyms such as “Lincoln”. DATE contributes seven tokens (5.9%), spanning both explicit references (e.g., “October 2018”) and phrasal expressions (e.g., “the fourth Friday of March”).