Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?

Momoka Furuhashi Kouta Nakayama Takashi Kodama Saku Sugawara, National Institute of Informatics National Institute of Informatics [email protected] {nakayama,tkodama,saku}@nii.ac.jp · 2025

Read Are Checklists Really Useful for Automatic Evaluation of Generative Tasks? by Momoka Furuhashi Kouta Nakayama Takashi Kodama Saku Sugawara, National Institute of Informatics National Institute of Informatics [email protected] {nakayama,tkodama,saku}@nii.ac.jp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15218.txt.

Description

Momoka Furuhashi, Kouta Nakayama, Takashi Kodama, and Saku Sugawara wrote this 2025 study on whether checklists help machines grade the output of generative AI. Grading such output is hard because the rules are often vague. Auto-made checklists look like a fix, but their worth had not been well tested.

The team made checklists by six methods and tried them across eight model sizes. Tests covered both head-to-head grading and direct scoring. Using checklists only for some questions tended to help head-to-head grading, while gains in direct scoring were mixed. Some checklist items that matched human scores poorly still echoed rules humans had written, which points to loose spots in human grading itself. The authors call for sharper grading rules for both people and machines.

Catalog record

CREATORMomoka Furuhashi Kouta Nakayama Takashi Kodama Saku Sugawara, National Institute of Informatics National Institute of Informatics [email protected] {nakayama,tkodama,saku}@nii.ac.jp
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15218

Browse the full library · About the Tower of Babel