What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning

Chi-Hsi Kung *, Frangil Ramirez *, Juhyung Ha, Yi-Ting Chen †, David J. Crandall †, Yi-Hsuan Tsai † · 2025

Read What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning by Chi-Hsi Kung *, Frangil Ramirez *, Juhyung Ha, Yi-Ting Chen †, David J. Crandall †, Yi-Hsuan Tsai † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21055.txt.

Description

Chi-Hsi Kung and co-authors teach video models about state changes in a 2025 paper on procedure-aware representation learning. Understanding a step-by-step activity means knowing how each action changes the scene, and how those changes shape what happens next, including mistakes.

The team uses large language models to write state-change descriptions for each step, then adds counterfactual versions that imagine how a step might fail. Video encoders train on both the real and imagined outcomes. Across six tasks, including temporal action segmentation, error detection, and frame retrieval, the approach brings clear gains. Learning what could have changed helps the model grasp cause and effect.

Catalog record

CREATORChi-Hsi Kung *, Frangil Ramirez *, Juhyung Ha, Yi-Ting Chen †, David J. Crandall †, Yi-Hsuan Tsai †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-21055

Browse the full library · About the Tower of Babel