Familiarity : Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data

Jonas Golde, Patrick Haller, Max Ploner, Fabio Barth, Nicolaas Jedema, Alan Akbik · 2024

Read Familiarity : Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data by Jonas Golde, Patrick Haller, Max Ploner, Fabio Barth, Nicolaas Jedema, Alan Akbik (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10121.txt.

Description

Jonas Golde and co-authors argue that scores for zero-shot named entity recognition are inflated, and they offer a metric called Familiarity to show by how much. Their 2024 paper deals with models that must spot entity types, such as persons or medicines, with no training examples of those types. Training often uses large synthetic datasets that cover thousands of types.

Those datasets often contain types nearly identical to the ones in standard test sets, so high scores overstate true skill. Familiarity measures the label shift between training and test sets, joining how alike the labels are with how often they appear. It lets researchers build test setups of graded difficulty and read zero-shot scores in proper context.

Catalog record

CREATORJonas Golde, Patrick Haller, Max Ploner, Fabio Barth, Nicolaas Jedema, Alan Akbik
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10121

Browse the full library · About the Tower of Babel