PRISM: P erception R easoning I nterleaved for S equential Decision M aking

Mohamed Salim Aissi, Clemence Grislain, Clement Romac, Laure Soulier, Mohamed Chetouani, Olivier Sigaud, Nicolas Thome · 2026

Read PRISM: P erception R easoning I nterleaved for S equential Decision M aking by Mohamed Salim Aissi, Clemence Grislain, Clement Romac, Laure Soulier, Mohamed Chetouani, Olivier Sigaud, Nicolas Thome (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05407.txt.

Introduction

Mohamed Salim Aissi and six co-authors present PRISM, a framework for agents that must see, think, and act. A vision-language model alone often misses details that matter for the task at hand. PRISM pairs that model with a language model in a loop of questions and answers.

The language model critiques the scene description and asks goal-focused questions. The vision model answers, and the pair builds a compact, task-driven picture of the scene. On the ALFWorld and Room-to-Room benchmarks, PRISM beat leading image-based systems. The gains came from the interactive questioning, with no hand-written questions needed.

Catalog record

CREATORMohamed Salim Aissi, Clemence Grislain, Clement Romac, Laure Soulier, Mohamed Chetouani, Olivier Sigaud, Nicolas Thome
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05407

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PRISM: P erception R easoning I nterleaved for S equential Decision M aking Mohamed Salim Aissi, Clemence Grislain, Clement Romac, Laure Soulier, Mohamed Chetouani, Olivier Sigaud, Nicolas Thome Abstract Scaling LLM-based embodied agents from text-only environments to complex multimodal settings remains a major challenge. Recent work identifies a perception–reasoning–decision gap in standalone Vision–Language Models (VLMs), which often overlook task-critical information. In this paper, we introduce prism , a framework that tightly couples perception (VLM) and decision (LLM) through a dynamic question–answer (DQA) pipeline. Instead of passively accepting the VLM’s description, the LLM critiques it, probes the VLM with goal-oriented questions, and synthesizes a compact image description. This closed-loop interaction yields a sharp, task-driven understanding of the scene. We evaluate prism on the ALFWorld and Room-to-Room (R2R) benchmarks. We show that: (1) prism significantly outperforms state-of-the-art image-based models, (2) our Interactive goal-oriented perception pipeline yields systematic and substantial gains, and (3) prism is fully automatic, eliminating the need for handcrafted questions or answers. 1 Introduction Figure 1 : Illustration of prism ’s interleaved perception (VLM) and reasoning (LLM). Given an image and goal, prism ’s dynamic question-answering (DQA) enables the LLM to query the VLM for task-critical information. This yields fine-grained, task-driven descriptions (bottom), ensuring accurate action prediction. Conversely, methods decoupling the VLM and LLM yield task-agnostic, incomplete descriptions, causing the agent to fail. The field of Embodied Artificial Intelligence, which aims to develop agents capable of perceiving and autonomously interacting with the physical world, has attracted significant attention since the emergence of foundation models. In particular, Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated remarkable emergent capabilities in logical reasoning, semantic understanding, and common-sense knowledge ( 12 ; 33 ; 15 ) . This naturally led to growing interest in studying how to integrate them as components in embodied agents that interact with the world. Our work studies how to incorporate them as components of agents interacting with visual environments in order to solve natural language instructions. Prior research has shown that LLMs can be leveraged for sequential decision-making when provided with a textual description of the environment, either zero-shot ( 32 ; 24 ) , or through fine-tuning ( 7 ; 27 ) . These approaches typically assume perfect perception, where prompts contain all necessary information. However, transitioning from such idealized text to raw visual input remains a major challenge. The impressive ability of VLMs to process images and generate text naturally positions them as candidates for this challenge ( 2 ; 18 ) . Recent works have explored two primary integration strategies: (1) directly replacing the LLM with a VLM ( 33 ; 31 ) , or (2) employing the VLM as an intermediate perception module to generate textual descriptions for a decision-making LLM ( 21 ; 37 ; 1 ) . However, performance still lags behind that achieved in textual environments. For the latter, this gap highlights the limited ability of VLMs to capture task-critical, fine-grained visual information when perception (VLM) and decision-making (LLM) are loosely coupled. In contrast, perception and decision-making are deeply intertwined in humans, where sensory inputs prioritize information essential to the current goal ( 10 ) . To mirror this interactivity, recent works have proposed introducing feedback mechanisms from decision-making to perception, using human clarification or question-answering ( 9 ; 23 ; 19 ) . However, the integration of perception and reasoning for decision-making is achieved through naive solutions, e.g., mere concatenation of textual interactions. More importantly, these approaches rely on predefined questions or answers, restricting their applicability to realistic settings. In this paper, we introduce prism 1 1 1 Code can be found here : P erception and R easoning I nterleaved for S equential Decision- M aking. prism establishes a tight coupling between perception and decision-making through a dynamic question-answering (DQA) mechanism. The perception module (VLM) provides an initial scene description, which the decision module (LLM) critiques against the goal to identify and query missing information. Once the VLM responds, the LLM synthesizes the interaction feedback into a compact, task-driven description. As illustrated in Figure 1 , prism ’s interleaved reasoning yields superior task-oriented descriptions compared to approaches decoupling VLM and LLM modules ( 37 ; 21 ; 1 ) . Our contributions can be summarized as follows: \bullet Dynamic Question-Answering (DQA): DQA tightly couples perception and decision-making, and synthesizes interaction feedback into a compact textual scene description. Crucially, prism ’s DQA is fully automatic, eliminating the need for unrealistic handcrafted or oracle annotations. \bullet Efficient training with prism ’s architecture: Separating perception via DQA from decision via the LLM enables the use of established recipes for text-based Reinforcement Learning (RL), while effectively grounding decisions through online interactions within the visual environment. \bullet Extensive experiments on ALFWorld ( 25 ) and Room-to-Room (R2R) ( 3 ) show that prism significantly outperforms current image-based models. We demonstrate consistent gains from DQA across both benchmarks and provide rich analyses validating the generated questions, answers, and descriptions. Figure 2 : The PRISM Framework: The agent processes a goal g and image observation o^{t} to execute an action a^{t} . (a) Interactive Goal-Oriented Perception: A VLM and LLM collaborate through dynamic question answering (DQA) to refine scene understanding: (1) the VLM provides an initial description d_{i}^{t} ; (2) the LLM generates goal-relevant questions \cal{Q} t ; (3) the VLM provides a specific answer \lambda_{i}^{t} for each question q_{i}^{t}\in \cal{Q} t ; and (4) the LLM synthesizes these into a final description d_{f}^{t} . (b) Decision-Making: The same LLM, augmented with LoRA adapters trained for action generation, takes d_{f}^{t} , the goal g , and the history \cal{H} to generate the action a^{t} executed in the environment. 2 Related Work 2.1 Visual sequential decision-making To reach goals from visual inputs, VLM-based agents have been explored. As zero-shot generalist VLMs often lack sufficient grounding, recent research has shifted toward RL fine-tuning, e.g., RL4VLM 33 . However, these agents significantly underperform LLMs provided with perfect textual descriptions 31 , as discussed in introduction and demonstrated in our experiments (Section 4 ). This disparity highlights the limited decision-making capacity of current VLMs and the difficulty of directly applying vanilla RL—highly effective for text-based inputs 7 ; 27 —to visual sequential decision-making. Further efforts have been made to design architectures combining VLMs and LLMs to leverage their complementary strengths. Typically, the VLM describes the visual scene while the LLM selects actions. Such systems have shown promise in navigation ( 3 ; 37 ; 21 ; 36 ) and interactive environments, e.g., VIPER ( 1 ) . However, perception and decision-making remain largely decoupled in these approaches: the VLM operates independently of the agent’s reasoning, often overlooking task-relevant visual details (see Figure 1 ). We adopt this modular architecture with a VLM for perception and an LLM for action selection, but introduce a dynamic question-answering (DQA) mechanism that tightly couples perception and decision-making. 2.2 Interactive question-answering in embodied agents Asking questions is a powerful mechanism to acquire missing information ( 26 ) , enabling agents to refine their environmental understanding for effective sequential decision-making. Prior work enabled agents to query external sources such as humans or oracles as in DialFRED ( 9 ) or ( 20 ; 8 )

Read the complete text