Generalizing Verifiable Instruction Following
Valentina Pyatkin Saumya Malik Victoria Graf, Hamish Ivison Shengyi Huang Pradeep Dasigi Nathan Lambert Hannaneh Hajishirzi · 2025
Read Generalizing Verifiable Instruction Following by Valentina Pyatkin Saumya Malik Victoria Graf, Hamish Ivison Shengyi Huang Pradeep Dasigi Nathan Lambert Hannaneh Hajishirzi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02833.txt.
Description
Valentina Pyatkin and co-authors study how well language models follow output constraints that can be checked by rule, such as limits on form or required words. Even strong models fail such precise instructions. Worse, models seem to overfit the few constraint types used in current tests and fail on new ones.
The team built IFBench, a test of fifty-eight new, varied and hard verifiable constraints from outside the usual domains. They also studied what training helps a model generalise. Reinforcement learning with verifiable rewards gave a clear lift in precise instruction following. They share twenty-nine new training constraints with checkers, plus prompts and code, with the benchmark.
Catalog record
| CREATOR | Valentina Pyatkin Saumya Malik Victoria Graf, Hamish Ivison Shengyi Huang Pradeep Dasigi Nathan Lambert Hannaneh Hajishirzi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-02833