Assessing the Reliability and Validity of GPT-4 in Annotating Emotion Appraisal Ratings

Deniss Ruder, Andero Uusberg, Kairit Sirts · 2025

Read Assessing the Reliability and Validity of GPT-4 in Annotating Emotion Appraisal Ratings by Deniss Ruder, Andero Uusberg, Kairit Sirts (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16883.txt.

Description

Deniss Ruder, Andero Uusberg, and Kairit Sirts test GPT-4 as a stand-in reader who rates emotional appraisals. Appraisal theory holds that feelings grow from how people judge events. Human raters score those judgments on scales for research data sets. The team had GPT-4 rate twenty-one appraisal scales under varied prompts.

The model matched or slightly beat human raters at the task. Averaging five runs by majority vote lifted its scores a good deal more. One prompt could yield both appraisal ratings and emotion labels well. Fancier instructions hurt rather than helped its marks. Longer event tales improved accuracy for model and humans alike.

Catalog record

CREATORDeniss Ruder, Andero Uusberg, Kairit Sirts
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16883

Browse the full library · About the Tower of Babel