What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning
Chi-Hsi Kung *, Frangil Ramirez *, Juhyung Ha, Yi-Ting Chen †, David J. Crandall †, Yi-Hsuan Tsai † · 2025
Read What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning by Chi-Hsi Kung *, Frangil Ramirez *, Juhyung Ha, Yi-Ting Chen †, David J. Crandall †, Yi-Hsuan Tsai † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21055.txt.
Description
Chi-Hsi Kung and co-authors teach video models about state changes in a 2025 paper on procedure-aware representation learning. Understanding a step-by-step activity means knowing how each action changes the scene, and how those changes shape what happens next, including mistakes.
The team uses large language models to write state-change descriptions for each step, then adds counterfactual versions that imagine how a step might fail. Video encoders train on both the real and imagined outcomes. Across six tasks, including temporal action segmentation, error detection, and frame retrieval, the approach brings clear gains. Learning what could have changed helps the model grasp cause and effect.
Catalog record
| CREATOR | Chi-Hsi Kung *, Frangil Ramirez *, Juhyung Ha, Yi-Ting Chen †, David J. Crandall †, Yi-Hsuan Tsai † |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21055