How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study

Zhen Yang, Ping Jian, Zhongbin Guo, Zuming Zhang, Chengzhi Li, Yonghong Deng, Xinyue Zhang, Wenpeng Lu · 2026

Read How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study by Zhen Yang, Ping Jian, Zhongbin Guo, Zuming Zhang, Chengzhi Li, Yonghong Deng, Xinyue Zhang, Wenpeng Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15294.txt.

Description

Yang and seven colleagues probe whether language models can grasp viewpoint rotation from text alone, without any images. Asked to track a viewpoint through described rotations and predict what would be seen, both language and vision-language models score poorly where people score perfectly. Layer probes show the models encode viewpoints but fail to bind them to observations, and tuning only the key attention heads improves scores without erasing general skills.

Catalog record

CREATORZhen Yang, Ping Jian, Zhongbin Guo, Zuming Zhang, Chengzhi Li, Yonghong Deng, Xinyue Zhang, Wenpeng Lu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58283
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15294

Browse the full library · About the Tower of Babel