Planning with Unified Multimodal Models

Yihao Sun, Zhilong Zhang, Yang Yu, Pierre-Luc Bacon · 2025

Read Planning with Unified Multimodal Models by Yihao Sun, Zhilong Zhang, Yang Yu, Pierre-Luc Bacon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23014.txt.

Description

Yihao Sun and 3 co-authors plan robot actions with unified multimodal models that both take in and produce images and text. Most prior planners reason in words alone, which limits what they can foresee. Their framework, Uni-Plan, lets one model play three roles at once. It acts as the policy, the world dynamics model, and the value judge.

Imagined futures can contain fake dynamics, so they add a guard. The same model grades its own imagined steps and filters out invalid ones. On embodied tasks, Uni-Plan raised success rates well past vision-language baselines. It needed no expert demos and scaled well with more training data.

Catalog record

CREATORYihao Sun, Zhilong Zhang, Yang Yu, Pierre-Luc Bacon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23014

Browse the full library · About the Tower of Babel