ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models

Dohwan Ko Sihyeon Kim Yumin Suh Vijay Kumar B.G, Minseo Yoon Manmohan Chandraker Hyunwoo J. Kim · 2025

Read ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models by Dohwan Ko Sihyeon Kim Yumin Suh Vijay Kumar B.G, Minseo Yoon Manmohan Chandraker Hyunwoo J. Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19355.txt.

Description

Programs that answer questions about video often stumble when asked how fast something moved or how far it went. Dohwan Ko and six co-authors tackle that gap in a 2025 paper. They built a training set from real videos with three-dimensional notes on motion, covering distance traveled, speed, direction, and how objects move relative to each other. A second pipeline labels ordinary videos automatically using rebuilt 3D scenes.

Trained on this data, their model beat rivals on their new test set and on older video benchmarks too. The authors show the skill also supports harder, multi-step questions about moving scenes.

Catalog record

CREATORDohwan Ko Sihyeon Kim Yumin Suh Vijay Kumar B.G, Minseo Yoon Manmohan Chandraker Hyunwoo J. Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-19355

Browse the full library · About the Tower of Babel