Familiarity : Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data
Jonas Golde, Patrick Haller, Max Ploner, Fabio Barth, Nicolaas Jedema, Alan Akbik · 2024
Read Familiarity : Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data by Jonas Golde, Patrick Haller, Max Ploner, Fabio Barth, Nicolaas Jedema, Alan Akbik (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10121.txt.
Description
Jonas Golde and co-authors argue that scores for zero-shot named entity recognition are inflated, and they offer a metric called Familiarity to show by how much. Their 2024 paper deals with models that must spot entity types, such as persons or medicines, with no training examples of those types. Training often uses large synthetic datasets that cover thousands of types.
Those datasets often contain types nearly identical to the ones in standard test sets, so high scores overstate true skill. Familiarity measures the label shift between training and test sets, joining how alike the labels are with how often they appear. It lets researchers build test setups of graded difficulty and read zero-shot scores in proper context.
Catalog record
| CREATOR | Jonas Golde, Patrick Haller, Max Ploner, Fabio Barth, Nicolaas Jedema, Alan Akbik |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10121