Human-in-the-Loop Nugget Annotation for Accountable LLM-as-a-Judge Evaluations

Laura Dietz · 2026

Read Human-in-the-Loop Nugget Annotation for Accountable LLM-as-a-Judge Evaluations by Laura Dietz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29033.txt.

Description

Dietz presents a prototype tool that splits the work of judging AI outputs between humans and language models in a way meant to keep the human signal real. Reliable evaluation needs human judgment, but common setups either anchor experts, nudging them to rubber-stamp model verdicts, or leave them unsupported in noisy labelling tasks where answers vary wildly.

Her design gives each side the job it does best. Humans identify nuggets, the pieces of information that matter for a good answer. Language models then handle the high-volume work of matching those nuggets against many system outputs. The workflow runs in three phases, and the exported nugget banks plug into automated judges, scaling the human decisions across large test sets. Dietz describes the key design choices behind the split. The aim is genuine oversight: accountability stays with the humans, while the tedious matching is automated.

Catalog record

CREATORLaura Dietz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29033

Browse the full library · About the Tower of Babel