Thinking in Text and Images: Interleaved Vision–Language Reasoning Traces for Long-Horizon Robot Manipulation

Jinkun Liu, Haohan Chi, Lingfeng Zhang, Yifan Xie, YuAn Wang, Long Chen, Hangjun Ye, Xiaoshuai Hao, Wenbo Ding · 2026

Read Thinking in Text and Images: Interleaved Vision–Language Reasoning Traces for Long-Horizon Robot Manipulation by Jinkun Liu, Haohan Chi, Lingfeng Zhang, Yifan Xie, YuAn Wang, Long Chen, Hangjun Ye, Xiaoshuai Hao, Wenbo Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00438.txt.

Description

Jinkun Liu and 8 co-authors teach robot policies to think in text and images at once. Long manipulation tasks need plans that are logically sound and geometrically grounded. Text-only reasoning orders the steps but misses spatial limits, while image prediction shows geometry but stays local and vague. Latent plans, hidden inside the network, cannot be inspected at all.

Their framework builds an explicit trace that alternates written subgoals with visual keyframes across the whole task. One multimodal model generates the trace from the opening view and the instruction, caches it, and a closed-loop controller then acts under its guidance. Training traces are manufactured by slicing demonstrations into stages and captioning each stage automatically. On long-horizon simulation benchmarks the interleaved trace far outperformed text-only, image-only, and no-trace variants, though the authors note stale or wrong global plans remain a weakness.

Catalog record

CREATORJinkun Liu, Haohan Chi, Lingfeng Zhang, Yifan Xie, YuAn Wang, Long Chen, Hangjun Ye, Xiaoshuai Hao, Wenbo Ding
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00438

Browse the full library · About the Tower of Babel