PSALM-V: Automating Symbolic Planning in Interactive Visual Environments with Large Language Models

Wang Bill ZhuMiaosen ChaiIshika Singh Robin Jia Jesse Thomason Department of Computer ScienceUniversity of Southern California · 2025

Read PSALM-V: Automating Symbolic Planning in Interactive Visual Environments with Large Language Models by Wang Bill ZhuMiaosen ChaiIshika Singh Robin Jia Jesse Thomason Department of Computer ScienceUniversity of Southern California (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20097.txt.

Introduction

Wang Bill Zhu and four co-authors presented PSALM-V in twenty twenty-five, a system that learns the rules of actions in visual worlds by trial and error. Symbolic planners need each action's preconditions and effects written in the PDDL language; hand-writing them is the bottleneck. Language models can draft them, but earlier attempts assumed text worlds, full observability, or explicit error messages.

PSALM-V instead interacts with the environment: it proposes plans and candidate semantics, executes, and explains failures by refining a tree of beliefs about each action's rules. It even infers the problem file itself. On the ALFRED household benchmark it raised plan success from thirty-seven to seventy-four percent under partial observability. It also induced correct rules for two games and a real robot block-stacking task.

Catalog record

CREATORWang Bill ZhuMiaosen ChaiIshika Singh Robin Jia Jesse Thomason Department of Computer ScienceUniversity of Southern California
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20097

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PSALM-V: Automating Symbolic Planning in Interactive Visual Environments with Large Language Models Wang Bill ZhuMiaosen ChaiIshika Singh Robin Jia Jesse Thomason Department of Computer ScienceUniversity of Southern California Abstract We propose PSALM-V, the first autonomous neuro-symbolic learning system able to induce symbolic action semantics (i.e., pre- and post-conditions) in visual environments through interaction. PSALM-V bootstraps reliable symbolic planning without expert action definitions, using LLMs to generate heuristic plans and candidate symbolic semantics. Previous work has explored using large language models to generate action semantics for Planning Domain Definition Language (PDDL)-based symbolic planners. However, these approaches have primarily focused on text-based domains or relied on unrealistic assumptions, such as access to a predefined problem file, full observability, or explicit error messages. By contrast, PSALM-V dynamically infers PDDL problem files and domain action semantics by analyzing execution outcomes and synthesizing possible error explanations. The system iteratively generates and executes plans while maintaining a tree-structured belief over possible action semantics for each action, iteratively refining these beliefs until a goal state is reached. Simulated experiments of task completion in ALFRED demonstrate that PSALM-V increases the plan success rate from 37% (Claude-3.7) to 74% in partially observed setups. Results on two 2D game environments, RTFM and Overcooked-AI, show that PSALM-V improves step efficiency and succeeds in domain induction in multi-agent settings. PSALM-V correctly induces PDDL pre- and post-conditions for real-world robot BlocksWorld tasks, despite low-level manipulation failures from the robot . Videos and resources at https://psalmv.github.io/ . Keywords: Vision Language Models, Neuro-symbolic, Symbolic Planning, PDDL 1 Introduction Planning in interactive, visual environments presents a fundamental challenge for embodied artificial intelligence. Symbolic planners have traditionally been a powerful tool for structured decision-making, generating sequences of actions to transition from an initial state to a goal state. These planners rely on expert-defined action semantics, typically encoded in domain-specific languages such as the Planning Domain Definition Language (PDDL; [ 1 ] ). Constructing these logical action semantics requires extensive manual effort and domain knowledge, limiting scalability to new environments. Recent LLM and robot research has explored leveraging large language models (LLMs) to alleviate the need for human-authored action semantics by generating PDDL-like action definitions [ 2 , 3 ] . While promising, these existing approaches have strong and unrealistic assumptions, such as: \bullet Focus on text-based domains : Real-world scenarios often require visual interaction to understand environmental dynamics. \bullet Reliance on full observability : Many real-world scenarios involve partial observability, where agents must infer missing information from past interactions. \bullet Predefined, fully observed symbolic initial state : While PDDL problem files characterizing the initial state of the world are provided in some environments, generating them, particularly in visual environments, is complex. \bullet Explicit error messages : Error messages are usually not available in real-world exploration. Previous work by Zhu et al. 2024 highlights failures without explicit error messages. \bullet List structure of pre- and post-conditions : The assumption that action semantics are intersections of logical conditions joined by and limits the use of other logical operators like or and when . To address these limitations, we introduce Predicting Semantics of Actions with Language Models for Vision and Robotics (PSALM-V), the first neuro-symbolic framework that tackles key challenges in automated planning through flexible semantic updates and dynamic trajectory sampling. Starting with a problem file initialized from visual input, PSALM-V explores partially observed environments, constructing semantic maps to aid problem file generation. The method iteratively samples and executes trajectories, predicts error messages, and updates action semantics to refine memory. It maintains a tree-structured belief over possible action semantics for each action, refining these beliefs until a valid plan is executed to achieve the goal state. A PDDL domain file, a symbolic representation of the rules of the world, is sampled based on the current memory of hypothesized action semantics, and a symbolic planner searches for a path to the goal state given those rules. We evaluate PSALM-V in three visual simulation environments and in a physical manipulation task with a Franka Panda arm. In simulation, we explore ALFRED [ 4 ] , RTFM [ 5 ] , and Overcooked-AI [ 6 ] . In ALFRED, PSALM-V increases the planning success rate from 37% (Claude-3.7) to 74% under conditions of partial observability. In RTFM and Overcooked-AI, PSALM-V improves step efficiency and successfully induces action semantics in multi-agent settings. Real-world BlocksWorld robot experiment results show that PSALM-V successfully recovers the full domain (F1 of 100%) with an average of 9.3 environment steps and achieves a goal-conditioned completion rate of 66.7%. Figure 1: LLMs can propose plans and generate action semantics, but struggle with state tracking. PDDL symbolic planners leverage specialized search algorithms, but require predefined action semantics for the environment. PSALM-V integrates the strengths of both for tasks on partially observed, visual environments. Objective Independent of Visual & Robot Partial AS Valid plans Human eval PF Full state Liu et al. 2023 PF ✗ ✓ ✓ ✗ ✗ ✗ Zhang et al. 2024 PF ✓ ✓ ✓ ✗ ✓ ✗ Lin et al. 2024 Plan ✓ ✓ ✗ - ✗ ✗ Silver et al. 2023 Program ✗ ✓ ✓ ✓ ✗ ✗ Arora et al. 2018 AS ✓ ✗ ✓ - ✗ ✗ Wong et al. 2023 AS ✗ ✓ ✓ ✓ ✗ ✗ Guan et al. 2023 AS ✓ ✓ ✗ ✓ ✗ ✗ Oswald et al. 2024 AS ✓ ✓ ✓ ✗ ✗ ✗ Zhu et al. 2024 AS ✓ ✓ ✓ ✗ ✗ ✗ Han et al. 2024 AS ✓ ✓ ✗ ✓ ✗ ✓ PSALM-V AS ✓ ✓ ✓ ✓ ✓ ✓ Table 1: PSALM-V domain induction task setup vs . representative related works in LLM and LLM-Modulo planning; here, “AS” is short for action semantics, “PF” is short for problem file, “Full state” means full state info is given. 2 Background and Related Works We first introduce symbolic planning, and then compare PSALM-V’s domain induction setup with previous LLM-augmented planning approaches, enumerated in Table 1 . 2.1 Classical and Symbolic Planning Classical planning algorithms have been extensively applied in domains such as autonomous spacecraft control, military logistics, manufacturing, games, and robotics. The automated STRIPS planner was the first algorithm to operate the Shakey robot [ 15 ] . Classical planners assume a finite, deterministic environment with full state observability, enabling them to generate guaranteed plans when a path from the initial to the goal state exists. Other planning frameworks, such as PRODIGY [ 16 ] and HTNs [ 17 ] , have also proven effective in robotic planning contexts. Symbolic planning languages, such as the Planning Domain Description Language (PDDL; [ 1 ] ) and Answer Set Programming (ASP; [ 18 , 19 ] ), offer structured and expressive ways to represent planning problems. We use PDDL for symbolic planning. We define a planning problem \mathrm{P} as a tuple \langle\mathcal{D},s^{i},\mathcal{S}^{g}\rangle , where \mathcal{D} denotes the domain, s^{i} the initial state, and \mathcal{S}^{g} the goal specification, i.e . , the set of goal states satisfying the defined goal conditions. A solution to \mathrm{P} is a T -step action sequence a_{1..T} such that executing a_{1..T} from s^{i} leads to a state in \mathcal{S}^{g} . The PDDL domain file defines the symbolic action set \mathcal{A} available in the environment,

Read the complete text