Do Language Models Track Entities Across State Changes?

Zilu Tang, Qiao Zhao, Gabriel Franco, Derry Wijaya, Aaron Mueller, Sebastian Schuster, Najoung Kim · 2026

Read Do Language Models Track Entities Across State Changes? by Zilu Tang, Qiao Zhao, Gabriel Franco, Derry Wijaya, Aaron Mueller, Sebastian Schuster, Najoung Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30233.txt.

Description

Zilu Tang and six co-authors ask whether language models track entities across state changes. Prior work studies entity binding without state changes, leaving realistic tracking unclear. They find models do not track world states step by step, but gather relevant information at the final token, processed in parallel once the query is clear. Probing single operations, they find REMOVE uses a fragile suppression tag, which predicts failure modes seen in behaviour. Removing that tag partly fixes the issue, and models solve a sequential task non-sequentially.

Catalog record

CREATORZilu Tang, Qiao Zhao, Gabriel Franco, Derry Wijaya, Aaron Mueller, Sebastian Schuster, Najoung Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103404
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30233

Browse the full library · About the Tower of Babel