Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversations

Michelle Elizabeth, Alicja Kasicka, Natalia Krawczyk, Magalie Ochs, Gwénolé Lecorvé, Justyna Gromada, Lina M. Rojas-Barahona · 2025

Read Neural Models and Language Model Prompting for the Multidimensional Evaluation of Open-Ended Conversations by Michelle Elizabeth, Alicja Kasicka, Natalia Krawczyk, Magalie Ochs, Gwénolé Lecorvé, Justyna Gromada, Lina M. Rojas-Barahona (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00841.txt.

Description

Michelle Elizabeth leads a team of seven authors on scoring open-ended conversations. Generative dialogue systems are growing fast in number. Judging their quality has become a crucial challenge. The work targets a Dialogue System Technology Challenge track. Models must give dialogue-level scores per dimension. All models used must stay under thirteen billion parameters.

The team pursues two strategies in parallel. One prompts language models to act as evaluators. The other trains encoder-based classification and regression models. Prompting achieves only modest agreement with human judgments. It still ranks second on the test set, behind only the baseline. The smaller trained models agree well on some validation dimensions. Their scores drop on the test set, though. Its score ranges differ sharply for some dimensions.

Catalog record

CREATORMichelle Elizabeth, Alicja Kasicka, Natalia Krawczyk, Magalie Ochs, Gwénolé Lecorvé, Justyna Gromada, Lina M. Rojas-Barahona
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-00841

Browse the full library · About the Tower of Babel