Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions

Seyedali Mohammadi, Bhaskara Hanuma Vedula, Hemank Lamba, Edward Raff, Ponnurangam Kumaraguru, Francis Ferraro, Manas Gaur · 2025

Read Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions by Seyedali Mohammadi, Bhaskara Hanuma Vedula, Hemank Lamba, Edward Raff, Ponnurangam Kumaraguru, Francis Ferraro, Manas Gaur (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02452.txt.

Description

Seyedali Mohammadi and six co-authors present a 2025 natural language processing paper asking whether language models truly use label definitions handed to them in a prompt. Models might instead fall back on knowledge baked in during training.

Controlled tests span general and domain-specific explanation datasets. Definition conditions include expert-written versions, model-generated ones, perturbed versions with small corruptions, and swapped versions taken from other labels. Clear definitions did raise accuracy and explanation quality in places. But uptake was uneven and never guaranteed, with models often defaulting to internal habits on general tasks. Domain tasks gained more from supplied definitions. The authors argue that how models blend given knowledge with stored knowledge needs deeper study.

Catalog record

CREATORSeyedali Mohammadi, Bhaskara Hanuma Vedula, Hemank Lamba, Edward Raff, Ponnurangam Kumaraguru, Francis Ferraro, Manas Gaur
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02452

Browse the full library · About the Tower of Babel