Integrating LMM Planners and 3D Skill Policies for Generalizable Manipulation

Yuelei Li, Ge Yan, Annabella Macaluso, Mazeyu Ji, Xueyan Zou, Xiaolong Wang · 2025

Read Integrating LMM Planners and 3D Skill Policies for Generalizable Manipulation by Yuelei Li, Ge Yan, Annabella Macaluso, Mazeyu Ji, Xueyan Zou, Xiaolong Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18733.txt.

Description

Yuelei Li and co-authors present a framework that lets a large multimodal model plan robot tasks while 3D skill policies carry them out. Big vision-language models reason well about scenes, and 3D feature fields give precise spatial grounding. Joining the two has been the hard part. Their system, LMM-3DP, works on three levels: high-level planning, low-level control, and the link between them.

For planning, the system understands changing scenes, critiques its own plans, remembers past policies, and retries after failures. For control, it uses a semantic 3D feature field for accurate manipulation. Language embeddings of the plan are attended jointly with the 3D field inside a transformer, tying the levels together. Tests across skills and long tasks in a real kitchen show clear gains: about 1.45 times the success rate in control and about 1.5 times the planning accuracy of language-model baselines.

Catalog record

CREATORYuelei Li, Ge Yan, Annabella Macaluso, Mazeyu Ji, Xueyan Zou, Xiaolong Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18733

Browse the full library · About the Tower of Babel