ProxAnn : Use-Oriented Evaluations of Topic Models and Document Clustering

Alexander Hoyle, Lorena Calvo-Bartolomé, Jordan Boyd-Graber, Philip Resnik · 2025

Read ProxAnn : Use-Oriented Evaluations of Topic Models and Document Clustering by Alexander Hoyle, Lorena Calvo-Bartolomé, Jordan Boyd-Graber, Philip Resnik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00828.txt.

Description

Alexander Hoyle and three co-authors build a fairer way to judge topic models. Their 2025 paper presents ProxAnn, an evaluation method grounded in real use. Automated scores for topic models track human taste poorly. Expert labels match better but cannot scale. The team wanted a test that mirrors how practitioners actually use these models.

In their protocol, a judge reads a topic's texts, names the category, then applies that name to new texts. Crowd workers ran the protocol on many models across two datasets. The team then checked whether language models could stand in for the humans. The best stand-ins matched a human judge closely enough to serve. Software, a web tool, and the data are shared freely.

Catalog record

CREATORAlexander Hoyle, Lorena Calvo-Bartolomé, Jordan Boyd-Graber, Philip Resnik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00828

Browse the full library · About the Tower of Babel