PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos

Meng Cao, Haoran Tang, Haoze Zhao, Hangyu Guo, Jiaheng Liu, Ge Zhang, Ruyang Liu, Qiang Sun, Ian Reid, Xiaodan Liang · 2024

Read PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos by Meng Cao, Haoran Tang, Haoze Zhao, Hangyu Guo, Jiaheng Liu, Ge Zhang, Ruyang Liu, Qiang Sun, Ian Reid, Xiaodan Liang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01800.txt.

Description

Meng Cao and colleagues introduce PhysGame, a benchmark for testing whether video language models understand physical commonsense. Gameplay videos often contain glitches that defy everyday physics. Those glitches make the videos a natural test of whether a model notices when the visual world breaks its own rules.

PhysGame contains eight hundred eighty videos of glitches spanning mechanics, kinematics, optics, and material properties, covering twelve kinds of physical commonsense. Evaluating current video models, the team found that open-source systems lag well behind proprietary ones. To close the gap they built PhysInstruct, an instruction-tuning dataset with more than one hundred forty thousand question-answer pairs. They also built PhysDPO, a preference dataset designed to resist shortcuts such as relying on misleading titles or too few frames. The benchmark gives researchers a concrete way to measure and improve physical reasoning in video models.

Catalog record

CREATORMeng Cao, Haoran Tang, Haoze Zhao, Hangyu Guo, Jiaheng Liu, Ge Zhang, Ruyang Liu, Qiang Sun, Ian Reid, Xiaodan Liang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01800

Browse the full library · About the Tower of Babel