Generalizing Verifiable Instruction Following

Valentina Pyatkin Saumya Malik Victoria Graf, Hamish Ivison Shengyi Huang Pradeep Dasigi Nathan Lambert Hannaneh Hajishirzi · 2025

Read Generalizing Verifiable Instruction Following by Valentina Pyatkin Saumya Malik Victoria Graf, Hamish Ivison Shengyi Huang Pradeep Dasigi Nathan Lambert Hannaneh Hajishirzi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02833.txt.

Description

Valentina Pyatkin and co-authors study how well language models follow output constraints that can be checked by rule, such as limits on form or required words. Even strong models fail such precise instructions. Worse, models seem to overfit the few constraint types used in current tests and fail on new ones.

The team built IFBench, a test of fifty-eight new, varied and hard verifiable constraints from outside the usual domains. They also studied what training helps a model generalise. Reinforcement learning with verifiable rewards gave a clear lift in precise instruction following. They share twenty-nine new training constraints with checkers, plus prompts and code, with the benchmark.

Catalog record

CREATORValentina Pyatkin Saumya Malik Victoria Graf, Hamish Ivison Shengyi Huang Pradeep Dasigi Nathan Lambert Hannaneh Hajishirzi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02833

Browse the full library · About the Tower of Babel