VLM-DEWM: Dynamic External World Model for Verifiable and Resilient Vision-Language Planning in Manufacturing Journal: Robotics and Computer-Integrated Manufacturing Guoqin Tang, Qingxuan Jia, Gang Chen, Tong Li, Zeyuan Huang, Ning Ji, Zihang Lv Abstract Vision-language model (VLM) shows promise for high-level planning in smart manufacturing, yet their deployment in dynamic workcells faces two critical challenges: (1) stateless operation—they cannot persistently track out-of-view states, causing world-state drift; and (2) opaque reasoning—failures are difficult to diagnose, leading to costly blind retries. This paper presents VLM-DEWM, a cognitive architecture that decouples VLM reasoning from world-state management through a persistent, queryable Dynamic External World Model (DEWM). Each VLM decision is structured into an Externalizable Reasoning Trace (ERT)—comprising action proposal, world belief, and causal assumption—which is validated against DEWM before execution. When failures occur, discrepancy analysis between predicted and observed states enables targeted recovery instead of global replanning. We evaluate VLM-DEWM on multi-station assembly, large-scale facility exploration, and real-robot recovery under induced failures. Compared to baseline memory-augmented VLM systems, VLM-DEWM improves state-tracking accuracy from 56% to 93%, increases recovery success rate from below 5% to 95%, and significantly reduces computational overhead through structured memory. These results establish VLM-DEWM as a verifiable and resilient solution for long-horizon robotic operations in dynamic manufacturing environments. 1 Introduction Flexible manufacturing systems increasingly rely on autonomous robots to execute long-horizon tasks such as multi-station assembly, inter-station material handling, and inventory management [ 20 ] . Unlike single-station operations, these tasks span multiple work areas where the workspace changes dynamically due to inventory replenishment, human intervention, occlusions, and execution disturbances. Reliable operation therefore requires long-horizon planning across stations, persistent tracking of out-of-view states, and fast failure diagnosis and recovery to minimize downtime. Pre-trained vision–language models (VLMs) offer strong semantic understanding and task decomposition capabilities, making them attractive for high-level robotic planning. However, VLM-based planners face a fundamental responsibility mismatch in dynamic manufacturing workcells: they are required to both reason semantically and maintain long-horizon world states within a transient context window. This coupling is inherently fragile—cross-station consistency degrades under occlusion and inventory changes, leading to world-state drift over time [ 5 , 9 ] . Moreover, when execution fails due to stale beliefs, the system lacks interpretable root-cause explanations [ 2 ] , and recovery degenerates into blind retries or expensive global replanning, increasing cycle time and downtime. Consider a representative multi-station task: a mobile manipulator retrieves a Type-1 part from Station A, transports it to Station C, and completes assembly. The robot’s memory indicates one Type-1 part at Station A, but upon arrival perception detects two identical parts due to replenishment. Without resolving this memory–observation conflict, the robot may pick the wrong part or fail to terminate. If the part is later dropped at an unobserved location, recovery further requires diagnosing both the failure cause (e.g., grasp loss, collision, or actuator fault) and the updated part location; otherwise the system again resorts to blind retries or global replanning. In isolation, each of these issues—memory staleness, observation conflict, undiagnosed failure—might be recoverable through local corrections. However, long-horizon tasks amplify their impact in ways that single-station operations do not. First, errors accumulate: a single unresolved memory–observation conflict propagates incorrect beliefs to all downstream decisions, causing cascading failures that compound across subsequent stations. Second, retry costs escalate non-linearly: each blind retry consumes cycle time without addressing the root cause, and over a 50-step task, even a 5% per-step failure rate yields near-zero end-to-end success ( 0.95^{50}\approx 8\% ). Third, diagnosis grows intractable: as the action history lengthens, identifying which step introduced the error becomes increasingly difficult without structured state tracking. These compounding effects point to three requirements for reliable long-horizon VLM planning: state persistence across stations and occlusions, decision verifiability to arrest error propagation, and failure diagnosability to enable efficient recovery. Rather than relying on larger context windows, we address this mismatch through architectural separation: persistent state maintenance is delegated to an external, queryable world model, while the VLM focuses solely on semantic reasoning over structured state representations.This separation yields two design principles. First, decouple state from reasoning: maintain an explicit, queryable world state that is continuously synchronized with perception and remains accessible across station transitions, instead of implicitly encoding state in the context window. Second, make decisions verifiable and diagnosable: externalize VLM outputs into auditable artifacts, validate them against physical consistency before execution, and compare them with post-execution outcomes to enable informed recovery. Based on these principles, we propose VLM-DEWM (VLM with Dynamic External World Model), a cognitive architecture operating on a ‘‘Database--Transaction--Verification’’ paradigm. DEWM maintains a persistent, queryable database of semantic relations, 3D geometry, and execution history. 1 1 1 Our “world model” refers to a persistent state database for maintaining the current world state, distinct from learned dynamics models in model-based RL that predict P(s^{\prime}|s,a) , or manually engineered PDDL domains that encode symbolic preconditions. DEWM does not model dynamics; it maintains a synchronized snapshot that supports structured, verifiable zero-shot task decomposition. On top of DEWM, an “Externalizable Reasoning Trace (ERT)” structures each VLM decision into an action proposal, a world belief, and a causal assumption, all of which are validated against physical consistency before execution. A closed-loop cognitive pipeline synthesizes task-specific queries from DEWM, audits VLM outputs through multi-layer checks, and performs targeted failure diagnosis when discrepancies arise. We evaluate VLM-DEWM in simulation and real-robot experiments across three manufacturing-relevant scenarios: multi-station assembly under occlusions, large-scale facility exploration with spatial queries, and dynamic recovery under induced execution failures. Compared to state-of-the-art memory-augmented VLM systems, VLM-DEWM improves state-tracking accuracy from 56% to 93% in exploration tasks, achieves 95% recovery success rate in settings where the baselines rarely succeed, and reduces VLM inference overhead by over 70%. These results demonstrate that architectural separation of state maintenance from semantic reasoning substantially improves robustness for long-horizon robotic operations in dynamic manufacturing environments. The main contributions are: 1. Dynamic External World Model (DEWM): Resolves world-state drift by decoupling state maintenance from reasoning, ensuring persistent cross-station memory despite occlusions and transient context limitations. 2. Externalizable Reasoning Trace (ERT): Mitigates opaque reasoning by structuring VLM decisions into verifiable artifacts, enabling pre-execution physical checks to arrest error propagation. 3. Discrepancy-Driven Diagnosis: Eliminates costly blind retries via CS -anchored discrepancy analysis, which leverages prediction-observation mismatches to enable precise, targeted recovery. 2 Related Work As established in Section 1 , long-horizon robotic manipulation poses three core challenges: state persistence across occlusions and station transitions, decision verifiability for auditable execution, and failure diagnosability for efficient recovery. We review related work through these three lenses, examining how existing approaches in VLM-driven planning (Section 2.1 ), memory-augmented agents (Section 2.2 ), and scene representations (Section 2.3 ) address—or fail to address—each challenge. 2.1 VLM-Driven Robot Planning: From Generalization to Auditable Execution Research in VLM-driven planning generally falls into two paradigms: joint training (e.g., PaLM-E [ 8 ] , PaLi-X [ 28 ] ) and frozen deployment (e.g., RT-2 [ 4 ] , OpenVLA [ 15 ] ). The latter has recently expanded to scalable transformers such as \pi_{0} [ 3 ] and RDT-1B [ 19 ] , enabling end-to-end visuomotor policies that operate closer to the control interface. While these methods demonstrate impressive generalization, they face a critical bottleneck in manufacturing: state accountability. In production environments, execution must be auditable: when an assembly task fails (e.g., insertion jamming), operators need to trace whether the error arose from perceptual occlusion, geometric drift, or an invalid logical precondition. However, end-to-end models encode state implicitly in transient contexts, making root-cause attribution ill-posed without explicit state variables. Strategies like Inner Monologue [ 12 ] or ProgPrompt [ 22 ] attempt to externalize reasoning via textual logs, yet free-form language does not guarantee the metric determinism required for digital twins or MES integration. Consequently, purely VLM-based planners remain “open-loop” with respect to state auditability. 2.2 Memory-Augmented VLMs: Semantic Persistence without Metric Verification The implicit state encoding discussed above manifests as state forgetting —the inability to persistently track object states across occlusions and station transitions. To address this, recent works couple VLMs with external knowledge bases, exemplified by SAGE [ 18 ] and GEAR [ 21 ] , which employ Retrieval-Augmented Generation (RAG) to fetch context from semantic maps or vector databases. While RAG improves persistence [ 6 ] , it introduces a dual mismatch in geometric grounding and update latency. First, a fundamental geometric gap exists: RAG relies on vector-space similarity (e.g., cosine distance), which cannot verify metric spatial relations required for physical manipulation. Retrieving a statement like “Part A is aligned” is insufficient to certify feasibility without explicit pose verification [ 1 ] . Even graph-based variants [ 17 ] abstract away metric details (e.g., object poses, spatial offsets) essential for collision-free planning. Second, a dynamic mismatch persists: the standard “update–index–retrieve” loop often lags behind fast-changing workcell states, causing retrieved context to reflect outdated information and leading to execution failures [ 16 ] . Agentic frameworks with self-correction (e.g., Corrective RAG [ 25 ] , Agent-G [ 11 ] ) partially address staleness but still lack geometry-grounded verification, limiting their applicability to long-horizon manipulation where persistent state tracking and metric validation are essential. 2.3 Scene Representations: Bridging Geometry and Assembly-Process-State Scene representations in robotics must balance geometric fidelity with semantic reasoning. Current approaches struggle to satisfy both demands simultaneously. On the geometric end, representations have evolved from discrete grids (OctoMap [ 10 ] ) to continuous neural fields (CATNIPS [ 7 ] , BundleSDF [ 24 ] ) and Gaussian splatting (GraspSplats [ 13 ] ). While achieving high visual fidelity, these methods typically treat assembly constraints as opaque optimization costs and lack explicit encoding of interaction phases—such as the transition from free-space motion to constrained insertion—creating barriers for failure-mode identification. On the semantic end, 3D scene graphs (e.g., DovSG [ 26 ] , RoboEXP [ 14 ] ) encode logical relations and action affordances, while methods like RP-SG [ 27 ] infer spatial relations under occlusion via graph networks. However, these approaches often discard metric details (e.g., precise 6-DoF poses) essential for physical feasibility verification and failure diagnosis. Existing frameworks rarely capture the complete picture: they either track geometry without process-aware semantics, or encode task logic without reliable metric grounding. VLM-DEWM addresses this gap through a hybrid \mathcal{S} - \mathcal{G} architecture that couples high-fidelity geometric tracking with diagnosable, process-aware state abstraction—providing both the metric substrate for physical verification and the semantic interface for VLM reasoning. 3 Methodology As argued in Section 1 , reliable VLM-based planning requires separating persistent state maintenance from semantic reasoning. We instantiate this principle through VLM-DEWM, a framework operating on a “Database–Transaction–Verification paradigm”: the DEWM serves as a persistent database maintaining world state, the VLM outputs an Externalizable Reasoning Trace (ERT) as a verifiable transaction, and a Verification Engine audits the ERT against DEWM before execution. Formally, the DEWM state at time t is defined as: \text{DEWM}_{t}=\langle\Omega_{t},M_{t},CS_{t}\rangle (1) where \Omega_{t}=\langle\mathcal{S},\mathcal{G},\mathcal{L}\rangle is the environment memory core (spatial network, semantic graph, and shape prior library), M_{t} is the task memory tracking execution progress, and CS_{t} is the constraint state anchoring the robot’s interaction phase for failure diagnosis. Figure 1 : Architectural Overview of the VLM-DEWM Framework. Operating on a Database-Transaction paradigm, the system is orchestrated through three tightly coupled modules. The perception module first projects raw multimodal sensory streams into a structured instantaneous snapshot ( Env_{t} ) via geometric projection and filtration. This snapshot drives the thinking module, where the VLM serves as the reasoning core to simultaneously update the external world model (DEWM)—the central knowledge hub comprising environmental ( \Omega ) and internal ( M,CS ) memory—and synthesize a structured External Reasoning Trace (ERT) . Crucially, the validation module treats the persistent memory in the DEWM as ground-truth evidence to rigorously verify the ERT’s logical and physical consistency before dispatching the final action sequence to the Robot Skill Library \mathcal{A} for execution. As illustrated in Fig. 1 , this formalism is instantiated as a closed-loop system with four modules: Perception , DEWM State Management , VLM-based Reasoning , and Validation . At each step, the system first perceives raw observations O_{t} and updates the world state \Omega together with an interaction-stage state CS . It then queries \Omega to construct task-relevant context for the VLM, which generates an ERT hypothesis for the next subtask. Before dispatching actions to the primitive skill library \mathcal{A} , the validation module checks the ERT against \Omega and blocks geometrically infeasible plans; after execution, the system compares the observed outcome with the ERT’s causal assumption to trigger diagnosis and informed replanning when discrepancies occur. 3.1 The Environment Memory Core ( \Omega ): A Hybrid Semantic-Geometric Representation The Dynamic External World Model (DEWM) serves as the system’s knowledge foundation. Its primary component, the Environment Memory Core \Omega , is designed based on the philosophy of structural decoupling : separating high-level semantic logic from low-level geometric precision. Formally, we define \Omega as a tripartite tuple: \Omega=\langle\mathcal{S},\mathcal{G},\mathcal{L}\rangle (2) where \mathcal{S} represents the physical state, \mathcal{G} encapsulates logical relations, and \mathcal{L} provides shape priors (see Fig. 2 ). Figure 2 : Structural Composition of the Environment Memory Core ( \Omega ). The Spatial Network ( \mathcal{S} ) maintains geometric state (left), while the Hierarchical Semantic Graph ( \mathcal{G} ) encodes logical relations (right). The Grounding Link ( \Phi_{ground} , red) anchors every semantic node to its physical instantiation. In this architecture, \mathcal{G} provides a queryable interface for VLM reasoning, \mathcal{S} serves as the metric substrate for feasibility checking and motion planning, and \mathcal{L} supplies on-demand high-fidelity geometry for robust 6-DoF registration and occlusion-robust shape inference. All downstream modules interact with \Omega via a semantic-to-metric grounding path : semantic queries on \mathcal{G} are resolved to geometric entities in \mathcal{S} via explicit links, enabling verification against metric constraints. 3.1.1 The Geometric Layer: Spatial Network ( \mathcal{S} ) The Spatial Network \mathcal{S} maintains the probabilistic state estimate of the physical state of the environment. To balance global planning efficiency with local manipulation precision, we adopt a dual-representation strategy: \mathcal{S}=\langle\mathcal{S}_{bg},\mathcal{S}_{obj}\rangle . This design emulates human spatial memory— “focus on foreground, blur the background” —where task-relevant objects receive high-fidelity tracking while static structures are compressed into occupancy grids. Specifically, entities segmented with task-relevant labels (Section 3.2.2 ) are registered in \mathcal{S}_{obj} , while unassociated geometric primitives are fused into \mathcal{S}_{bg} . 1. Global Background Map ( \mathcal{S}_{bg} ): We utilize a coarse-grained, semantically labeled probabilistic voxel grid to represent static structures (e.g., walls, workbenches). This provides the free-space information necessary for global collision avoidance and coarse motion planning. 2. Dynamic Object Map ( \mathcal{S}_{obj} ): This is an instance-indexed registry tracking all interactive entities. For each tracked object i , its geometric state s_{obj}^{i} is defined as: s_{obj}^{i}=\langle\text{id}_{i},\mathbf{T}_{i},\mathcal{E}_{i}\rangle (3) Here, \text{id}_{i} is a pointer to the Shape Library \mathcal{L} ; \mathbf{T}_{i}\in SE(3) denotes the precise 6-DoF pose updated via multi-frame tracking; and \mathcal{E}_{i}(\boldsymbol{\mu}_{i},\boldsymbol{\Sigma}_{i}) represents a Gaussian Envelope. Remark: The Gaussian envelope \mathcal{E}_{i} acts as a probabilistic spatial index for rapid VLM queries (e.g., “near X”) and candidate pruning, rather than for fine-grained contact modeling. When high-precision pose estimation is required (e.g., for grasping or insertion), precise geometry is retrieved on-demand from \mathcal{L} via the shape prior link. 3.1.2 The Semantic Layer: Hierarchical Semantic Graph ( \mathcal{G} ) The semantic layer \mathcal{G} serves as the interface for VLM reasoning. We construct a Hierarchical Semantic Graph (HSG) to support multi-resolution planning: \mathcal{G}=\langle\mathcal{G}_{zone},\mathcal{G}_{obj}\rangle . 1. Zone Graph ( \mathcal{G}_{zone} ): Acts as the spatial root, where nodes represent topological regions (e.g., Workbench_A ) and edges denote reachability, allowing the planner to efficiently prune the search space. 2. Object Graph ( \mathcal{G}_{obj} ): Modeled as a graph \mathcal{G}_{obj}=(\mathcal{V},\mathcal{E}) , where each vertex v_{i}\in\mathcal{V} represents an object instance: v_{i}=\langle\text{uid}_{i},\text{label}_{i},\text{state}_{i},\text{Attr}_{i},\Phi_{ground}\rangle (4) Here, \text{Attr}_{i} stores feature attributes extracted from VLM perception (e.g., “fragile”, “stack_on _X”). The function \Phi_{ground}:\mathcal{V}\to\mathcal{S}_{obj} explicitly links the semantic node to its geometric counterpart in \mathcal{S} , ensuring that every abstract concept is physically grounded. Edges e_{ij}\in\mathcal{E} encode spatial relations (e.g., On ) or affordances, derived from VLM inference. Note that these relations are stored as soft hypotheses and can be revised by the verification engine if inconsistent with geometric observations. 3.1.3 The Shape Prior Library ( \mathcal{L} ): Digital-Twin-Aligned Priors The library \mathcal{L} stores high-fidelity geometric priors derived from CAD models, providing digital-twin-aligned references for robust pose estimation and shape completion. In manufacturing settings, we pre-process CAD meshes into canonical point clouds associated with functional frames. These priors support robust 6-DoF registration under sparse visual features and occlusion-robust shape inference when objects are partially covered by the gripper. Detailed registration algorithms and occlusion handling strategies are provided in B.1 . 3.1.4 Dynamic Maintenance The mechanisms for updating \Omega from perception—including data association, state fusion, and lifecycle management—are detailed in Section 3.2 . 3.1.5 Formal State Transition Semantics To ensure the deterministic maintenance of the world model, we define the DEWM update as a state transition function \delta:(\Omega_{t},CS_{t})\times\pi_{act}\to(\Omega_{t+1},CS_{t+1}) . This transition is designed to approximate database-style atomicity: if the rollback trigger is activated, the state reverts to (\Omega_{t},CS_{t}) . Specifically, for a primitive action \pi_{act} , the update is an atomic transaction defined in Table 1 . Table 1 : Atomic State Transition Rules for Primitive Skills Action \pi Pre-Condition ( \mathcal{G} ) Atomic Effects ( \Omega_{t+1},CS_{t+1} ) Rollback Trigger Pick(obj) \exists\texttt{Clear(obj)} 1. \mathcal{S}_{obj} : Detach \mathcal{PC}_{obj} 2. \mathcal{G} : Del On(obj, *) 3. CS : Set Holding(obj) Gripper width > limit (Slip) Place(loc) CS=\texttt{Holding(obj)} 1. \mathcal{S}_{obj} : Anchor at loc 2. \mathcal{G} : Add On(obj, loc) 3. CS : Set Idle Force feedback > limit (Collision) Insert(A, B) \exists\texttt{Aligned(A, B)} 1. \mathcal{G} : Add Inserted(A, B) 2. CS : Set Idle Torque > limit / Stall Note: If the Rollback Trigger is activated via low-level controller feedback (see B.7 for the specific force-torque thresholds and event mapping logic), the state reverts to (\Omega_{t},CS_{t}) . This table presents representative primitives. The full skill library \mathcal{A} includes additional primitives such as Move(loc) , Rotate(obj, angle) , and Open/Close_Gripper , each with corresponding transition rules. 3.2 Perception and Memory Dynamics: Maintaining Model Veracity While Section 3.1 defined the static structure of the DEWM, this section details the dynamic processes that synchronize the internal model \Omega with the evolving physical world. We employ a closed-loop pipeline “ Perceive \rightarrow Associate \rightarrow Fuse \rightarrow Verify ” to transform raw sensory streams into structured, physically grounded beliefs(see Fig. 3 ). Figure 3 : The Perception-to-Memory Synchronization Pipeline. Raw observations ( O_{t} ) are transformed into persistent belief updates via three stages: (1) task-focused perception producing Env_{t} , (2) geometric-semantic data association, and (3) dual-layer updates (continuous S-Update and discrete G-Update with geometric verification). 3.2.1 Task-Focused Perception Interface Unlike passive perception, our system employs a VLM-driven attention mechanism. Given a task instruction, the VLM generates a set of Target Entities and Regions of Interest . Raw RGB-D streams O_{t} are processed to extract geometric primitives only for these relevant targets, lifting 2D masks into 3D space. Formally, this produces an instantaneous snapshot Env_{t}=\{o_{k}\}_{k=1}^{K} , where each observation o_{k} contains a semantic label, a confidence score, and a distance-adaptive geometric abstraction—dense point clouds for near-field manipulation, voxels for mid-range collision checking, or billboard textures for far-field situational awareness (detailed in A ). 3.2.2 VLM-Driven Data Association Integrating new observations requires robust data association. Purely geometric matching fails in cluttered scenes, while purely semantic matching is computationally prohibitive. We therefore adopt a “Filter-then-Adjudicate” protocol that combines geometric pre-screening with VLM-assisted global assignment. a. Geometric Gating (Filter): We prune infeasible pairs by computing the Mahalanobis distance d_{M} between the observation’s Gaussian envelope (\boldsymbol{\mu}_{obs},\boldsymbol{\Sigma}_{obs}) and the memory entry’s envelope \mathcal{E}_{i}=(\boldsymbol{\mu}_{i},\boldsymbol{\Sigma}_{i}) defined in Section 3.1 d_{M}(obs,i)=\sqrt{(\boldsymbol{\mu}_{obs}-\boldsymbol{\mu}_{i})^{\top}(\boldsymbol{\Sigma}_{obs}+\boldsymbol{\Sigma}_{i})^{-1}(\boldsymbol{\mu}_{obs}-\boldsymbol{\mu}_{i})}. (5) Pairs with d_{M}\geq\tau_{geo} are rejected, avoiding physically impossible matches before semantic evaluation. b. Semantic–Geometric Adjudication (Judge) On the remaining feasible pairs, we solve a global one-to-one assignment via the Hungarian algorithm using a hybrid cost: C_{ki}=\lambda_{iou}\bigl(1-\text{IoU}(o_{k},s_{i})\bigr)+\lambda_{sem}\bigl(1-\text{sim}(o_{k},s_{i})\bigr). (6) We implement geometric gating by masking infeasible edges, i.e., setting C_{ki}=+\infty for pairs failing d_{M}<\tau_{geo} . Here, IoU is computed over 3D oriented bounding volumes (with a 2D fallback when depth is unreliable), and \text{sim}(\cdot) denotes VLM-based semantic similarity. Matches must pass a final quality gate C_{ki}\leq\tau_{match} . Unmatched observations enter a short confirmation window before being promoted to active entries, suppressing one-off false detections and preventing ghost artifacts. All prompts, parsing rules, and association hyperparameters are provided in B.1 . 3.2.3 VLM-Modulated Gaussian Envelope Fusion (S-Update) Once associated, the state update is modulated by VLM reasoning. Standard filters degrade when semantic context affects reliability (e.g., transparent objects or gripper-induced occlusion). We therefore perform a semantically-adaptive Gaussian fusion. For matched pairs, we maintain a Gaussian belief (\boldsymbol{\mu}_{t},\boldsymbol{\Sigma}_{t}) representing the object’s spatial envelope. Given a new observation with parameters (\boldsymbol{\mu}_{obs},\boldsymbol{\Sigma}_{obs}) , we perform a weighted fusion in the information filter form: \displaystyle\boldsymbol{\Sigma}_{t}^{-1} \displaystyle=\boldsymbol{\Sigma}_{t-1}^{-1}+\gamma(\alpha_{t})\cdot\boldsymbol{\Sigma}_{obs}^{-1} (7) \displaystyle\boldsymbol{\mu}_{t} \displaystyle=\boldsymbol{\Sigma}_{t}\left(\boldsymbol{\Sigma}_{t-1}^{-1}\boldsymbol{\mu}_{t-1}+\gamma(\alpha_{t})\cdot\boldsymbol{\Sigma}_{obs}^{-1}\boldsymbol{\mu}_{obs}\right) (8) where \alpha_{t}\in\{\text{High, Medium, Low, Bad}\} is a discrete reliability judgment produced by the VLM from scene context (e.g., occlusion, lighting, and sensor quality), and \gamma(\alpha_{t}) maps this judgment to a continuous weight factor that modulates the contribution of the new observation. This formulation treats the VLM as a meta-cognitive controller: when the VLM judges the observation as “High” reliability, \gamma is large, allowing the new measurement to strongly influence the belief; when judged as “Bad” (e.g., due to heavy occlusion), \gamma approaches zero, effectively “freezing” the belief at its prior value. This enables context-aware state estimation that traditional adaptive filters cannot achieve—for instance, adopting a conservative update strategy for fragile objects even when observation quality appears high. The noise models, prompt templates, decoding/parsing rules, and the full mapping table used in all experiments are provided in B.2 . 3.2.4 Geometric Refinement and Background Maintenance For key objects requiring high-precision assembly, we perform a scale-aware robust ICP registration to align the current observation with the canonical Shape Prior \mathcal{L} , refining the pose estimate \mathbf{T}_{i}\in SE(3) (details in B.3 ). Simultaneously, unassociated points are fused into the global background map \mathcal{S}_{bg} using probabilistic occupancy updates, and open boundaries are handled via view-dependent billboard textures to provide visual context for navigation (details in B.4 ). 3.2.5 Hypothesis-Verification Loop (G-Update) To maintain semantic consistency, relations proposed by the VLM (e.g., On(A,B) ) are treated as hypotheses and verified against \mathcal{S} at action boundaries . For proximity-type relations (e.g., Contact( A,B ) ), we validate consistency via a Chi-square test on Gaussian position estimates: (\boldsymbol{\mu}_{B}-\boldsymbol{\mu}_{A})^{\top}(\boldsymbol{\Sigma}_{A}+\boldsymbol{\Sigma}_{B})^{-1}(\boldsymbol{\mu}_{B}-\boldsymbol{\mu}_{A})\leq\chi^{2}_{thresh}. (9) For support-type relations (e.g., On( A,B ) ), we additionally apply conservative bounding-volume support checks with tolerance \delta_{supp} as a proxy for physical support. The statistical basis, threshold selection, and verification criteria (including \delta_{supp} ) for common relations are provided in B.5 . This verification acts as a geometry-grounded firewall, preventing hallucinated relations from propagating into the planner’s worldview. 3.2.6 Long-Term Memory Curation To prevent memory saturation in long-horizon tasks, we implement region-based confidence decay and lifecycle management. When the robot scans a zone Z_{k} and fails to re-observe an expected entity v_{i} , its confidence score decays: c_{t}(v_{i})=c_{t-1}(v_{i})\cdot\lambda_{decay}. (10) Entities transition through Active / Uncertain / Archived states based on confidence thresholds, and archived entities can be restored when re-observed. The complete lifecycle policy and curation algorithm are provided in B.6 . 3.2.7 Commit Conditions and Failure Handling The state transition is committed if and only if (1) the physical action is successfully executed by the controller (verified via force-torque feedback), and (2) the post-action perception verifies the expected geometric change specified in the ERT’s causal_assumption field (Section 3.3.3 ). Specifically, for Pick(obj) , we verify that obj ’s position now tracks the gripper frame; for Place(loc) , we verify obj is stationary at loc within tolerance \epsilon_{place} ; for Insert(A,B) , we verify the relative pose \mathbf{T}_{AB} matches the expected mating geometry. If either condition fails (e.g., the object slips during transport), the transaction is rolled back and the Diagnosis Module (Section 3.3.5 ) is triggered, preventing the “hallucination accumulation” common in open-loop VLM planners. 3.3 VLM-Guided Planning, Execution, and Recovery Having established a dynamic world model ( \Omega ), we now detail the decision-making engine: the “Progressive VLM-Guided Planning Algorithm ( \Pi )”. As illustrated in Fig. 4 , this engine operates as a closed-loop controller that tightly couples VLM reasoning with the structured DEWM. Assuming a primitive skill library \mathcal{A} for low-level execution, the decision process follows a rigorous State-Input-Process-Output cycle to generate verifiable, adaptive behavior. Figure 4 : The Progressive VLM-Guided Planning and Deep Recovery Loop (Sec. 3.3). Dynamic Context Engineering distills the DEWM state ( \Omega,M,CS ) into a task-relevant prompt. The VLM then generates an External Reasoning Trace (ERT) which is mathematically validated before execution. On failure, the Deep Causal Diagnosis module computes the belief-outcome discrepancy ( \Delta=\Omega_{t+1}\ominus\mathcal{A}_{causal} ) to generate targeted feedback for belief correction and informed recovery. 3.3.1 Internal Working Memory: Tracking Task and Ego-States Robust long-horizon autonomy requires tracking not just the external environment, but also the agent’s internal progress. We introduce two dedicated working memory modules: a. Task Memory ( M_{t} ) This module serves as the “project manager”, tracking global operation progress. Formally, M_{t}=\langle\mathcal{T}_{DAG},ss_{t},\mathcal{H}_{ash}\rangle : 1. \mathcal{T}_{DAG} : The “Task Topology Path”, a Directed Acyclic Graph (DAG) decomposed from user instructions via VLM-based task decomposition (prompt template in C.2 ). 2. ss_{t} : A state pointer to the active node in \mathcal{T}_{DAG} (Status: Executing / Completed ). 3. \mathcal{H}_{ash} : The “Action Sequence History”, a structured log recording every executed ERT. This serves as the “episodic memory” enabling causal diagnosis. b. Constraint State ( CS_{t} ) The Constraint State CS_{t} is central to our diagnosis mechanism (Contribution 3). While \mathcal{S} tracks geometric configurations, CS_{t} anchors the robot’s semantic interaction phase (e.g., \langle\texttt{Holding},\text{blue\_block\_1}\rangle , or \langle\texttt{Approaching},\text{target\_zone}\rangle ). Updates to CS_{t} are event-driven, occurring only when the causal_assumption of a validated ERT is functionally fulfilled. This mechanism provides the micro-topological anchor required for precise failure localization: when an anomaly occurs, CS_{t} identifies which interaction phase failed, enabling targeted diagnosis rather than generic task-level error handling. 3.3.2 Dynamic Context Engineering VLMs are inherently context-agnostic. To ground their reasoning and prevent “lost-in-the-middle” hallucinations, we design a “Subgraph Extraction Pipeline” that acts as a deterministic filter between the DEWM and the VLM. Instead of feeding the entire world state \Omega , which introduces irrelevant noise, the mechanism utilizes the current subtask ss_{t} and topological state CS_{t} to index the Hierarchical Semantic Graph ( \mathcal{G} ). It extracts a “Task-Relevant Subgraph”—the minimal set of objects and relations required for the current decision—and serializes it into a concise natural language prompt. This strategy minimizes token usage while maximizing information density. The subgraph extraction algorithm is detailed in C.1 , with context-specific prompt templates for task decomposition ( C.2 ), motion planning ( C.3 ), and fine manipulation ( C.4 ). 3.3.3 The Cognitive Core: External Reasoning Trace (ERT) To transform the VLM from an opaque “black box” into a verifiable agent, we enforce a structured output format called the “External Reasoning Trace (ERT)”. Every planning step produces a JSON object containing four key fields: ERT=\{\pi_{act},\mathcal{B}_{world},\mathcal{A}_{causal},\kappa_{conf}\} (11) Here, \pi_{act} is the action proposal, \mathcal{B}_{world} represents structured assertions about the pre-condition, \mathcal{A}_{causal} encodes the expected post-conditions, and \kappa_{conf} is the self-evaluated confidence. Crucially, the ERT functions as a probabilistic hypothesis subject to verification. Before execution, it undergoes a Multi-Layered Validation Protocol (see D ), which rigorously checks the consistency between the VLM’s semantic beliefs ( \mathcal{B}_{world} ) and the geometric ground truth in DEWM. This mechanism acts as a cognitive firewall, intercepting hallucinations before they manifest as physical errors. It is important to note that the ERT protocol focuses exclusively on Semantic Task Feasibility (e.g., verifying geometric containment). We explicitly decouple low-level kinematics and dynamics—such as collision avoidance and singularity handling—to the downstream execution layer. Exceptions at this lower level are captured as feedback signals and routed to the Constraint State ( CS ) module to trigger the diagnosis pipeline. 3.3.4 Main Planning Loop and Adaptive Control The core control logic is encapsulated in the progressive planning algorithm (Algorithm 1 ). To ensure robustness, all inference calls are encapsulated within a “Verify-Retry” wrapper, GetValidERT . The system operates in two hierarchical modes: 1. Macro-Planning (Algorithm 1 ): Handles high-level state transitions and discrete single-step actions. 2. Micro-Routine (Algorithm 2 ): Synthesizes detailed primitive sequences for contact-rich manipulation. Algorithm 1 Progressive VLM-Guided Planning (Main Loop) 1: Task \mathcal{T} , Skill Library \mathcal{A} , Max Retries N_{max} 2: Task Status: Completed or Failed 3: \Omega,M,CS\leftarrow\textsc{InitializeSystem}(\mathcal{T}) 4: while \textsc{GetStatus}(M)\neq\texttt{Completed} do 5: \Omega_{t},M_{t},CS_{t}\leftarrow\textsc{PerceiveAndUpdate}(\Omega_{t-1}) 6: ss_{t}\leftarrow\textsc{GetCurrentSubtask}(M_{t}) 7: if ss_{t} is ManipulationTask then 8: status\leftarrow\textsc{ExecuteMicroRoutine}(\Omega_{t},M_{t},CS_{t},ss_{t}) 9: else 10: Prompt\leftarrow\textsc{SynthesizeContext}(\Omega_{t},M_{t},CS_{t}) 11: ERT,valid\leftarrow\textsc{GetValidERT}(Prompt,N_{max}) 12: if \neg valid then return Failed 13: end if 14: skill\leftarrow\textsc{ExtractAction}(ERT) 15: status\leftarrow\textsc{Execute}(skill,\mathcal{A}) 16: end if 17: \textsc{UpdateState}(M_{t},CS_{t},status,ERT) 18: if status==\texttt{Failed} then 19: \textsc{DeepCausalDiagnosis}(\Omega_{t},M_{t},ERT) \triangleright Trigger Recovery 20: end if 21: end while 22: return Completed Algorithm 2 Micro-Manipulation Subroutine 1: World \Omega_{t} , Memory M_{t} , State CS_{t} , Subtask ss_{t} 2: Prompt\leftarrow\textsc{SynthesizeMicroPrompt}(\Omega_{t},M_{t},CS_{t},ss_{t}) 3: ERT_{micro},valid\leftarrow\textsc{GetValidERT}(Prompt,N_{max}) 4: if \neg valid then return Failed 5: end if 6: Seq\leftarrow\textsc{ExtractSequence}(ERT_{micro}) 7: for skill in Seq do 8: status\leftarrow\textsc{Execute}(skill,\mathcal{A}) 9: \textsc{UpdateState}(M_{t},CS_{t},status,ERT_{micro}) 10: if status==\texttt{Failed} then return Failed 11: end if 12: end for 13: return Success 3.3.5 Deep Causal Diagnosis and Belief Correction ( CS -Anchored Recovery) When a subtask fails (Line 18, Algorithm 1 ), the recovery mechanism goes beyond simple retries. The Belief-Action History ( \mathcal{H}_{ash} ) retains the full ERT recorded prior to the failure. The DeepCausalDiagnosis function triggers a diagnostic flow by computing the semantic divergence \Delta between the causal_assumption in the ERT and the actual post-failure reality \Omega_{t+1} : \Delta=\Omega_{t+1}\ominus\mathcal{T}_{ert}.\mathcal{A}_{causal} (12) Here, \ominus denotes the set difference of semantic states (formally defined in C.6 ). This calculation generates a precise Belief-Outcome Discrepancy Report ( C.6 ), which is fed back to the VLM via SynthesizeReplanPrompt (template in C.5 ). This enables the agent to perform Belief Correction —updating its internal model based on evidence—rather than merely generating a different plan, fundamentally distinguishing our approach from blind retry strategies. 4 Experiments To rigorously validate the effectiveness of the proposed VLM-DEWM cognitive architecture, we designed a comprehensive suite of experiments across high-fidelity simulation and real-world physical platforms. Our evaluation aims to answer three core Research Questions (RQs): 1. Long-Horizon Reliability: Can DEWM provide persistent, cross-regional world memory to support state tracking and zero-shot spatial reasoning, outperforming pure VLMs that rely solely on context history? 2. Precise Diagnosis: Compared to other memory-enhanced strategies, does the Micro-Topological State Anchoring ( CS ) mechanism enable more accurate causal diagnosis and intelligent recovery in dynamic environments? 3. Sim-to-Real Generalization: Can the core cognitive logic of VLM-DEWM generalize from simulation to physical interaction with real-world sensor noise and latency? 4.1 Experimental Setup This section details our experimental configuration. We describe the baseline methods in Section 4.1.1 , platform implementation details in Section 4.1.2 , experimental assumptions in Section 4.1.3 , task design in Section 4.1.4 , and evaluation metrics in Section 4.1.5 . 4.1.1 Baseline Methods We compare VLM-DEWM against two categories of baselines representing mainstream embodied planning paradigms. a. Context-Only VLM Planners This category relies exclusively on the VLM’s context window to maintain world understanding, without explicit external memory structures. We include both SOTA closed-source models (Gemini 2.5 Pro, GPT-5) and open-source models (Qwen3-VL-32B-Instruct, GLM-4.5V). For fairness, all VLM baselines are integrated into the same perception-planning loop, with their “memory” consisting solely of historical images and text prompts. b. Memory-Enhanced Strategies This category constructs external memories for VLMs and is most relevant to our work. We select “SAGE” [ 18 ] (a 3D Scene Graph approach) and “GEAR” [ 21 ] (a SOTA VLM-RAG method) as representative competitors, reproducing them according to their official specifications. c. Ours (VLM-DEWM) We utilize Gemini 2.5 Pro as the core “cognitive collaborator”. This choice allows for a direct “apples-to-apples” comparison with the Pure Gemini 2.5 Pro baseline, isolating the performance gains attributable to the DEWM architecture rather than the underlying model capability. 4.1.2 Implementation Details a. Simulation Environment Our simulation platform is built on PyBullet and Gazebo. To rigorously isolate cognitive performance from perceptual noise, we adopt a "privileged information" strategy in simulation. Perfect segmentation masks (via p.getCameraImage ) and ground-truth 6D poses are fed directly to the cognitive module. This ensures that the evaluation strictly benchmarks the DEWM’s logic (memory, planning, diagnosis) rather than the upstream perception model errors. b. Real-World Platform and Protocols The physical validation was conducted on a high-fidelity robotic cell designed to emulate industrial assembly (Fig. 11 ). 1. Hardware Specs: We use a Franka Emika Panda (7-DoF, 1kHz control loop) equipped with a Robotiq 2F-85 gripper. Perception is provided by an Intel RealSense D435i (848 \times 480 resolution @ 30fps), mounted extrinsically. 2. Sample Size ( N ): For Task 3, we conducted a total of 60 trials among which 20 trials were subjected to induced failures (covering 3 failure modes: Part Slip, Obstacle, Target Moved). 3. Success Criteria: A trial is successful if and only if the final assembly is completed within a 5mm position tolerance and 2 degree orientation tolerance relative to the CAD specification, verified by a calibrated overhead camera. 4. Throughput Metric: We define Effective Throughput as \frac{3600s}{\text{Avg. Cycle Time}+\text{MTTR}} . This penalizes methods that require manual reset after failure. To filter perceptual noise, we adopt our prior Coarse-to-Fine 3D-Grounded Perception framework [ 23 ] , ensuring the evaluation focuses on cognitive performance rather than upstream perception errors. This framework integrates open-vocabulary 2D detection with 3D point cloud clustering and outlier removal. By enforcing both semantic consistency and geometric density constraints, it effectively filters out perceptual hallucinations before they enter the DEWM update loop. This allows our evaluation to focus specifically on the cognitive performance of the memory and reasoning modules. 4.1.3 Experimental Assumptions and Priors To ensure rigor and alignment with target applications (e.g., smart manufacturing), we define the following assumptions: 1. Semi-Structured Environment: We assume object categories are known, but their poses, quantities, and topological relationships are unknown and dynamic. 2. Shape Prior Library ( \mathcal{L} ): For high-precision interaction (Task 3), the system is provided with a Shape Prior Library containing CAD models and grasp priors, as described in Section 3.2 . 3. Fairness Note: This library \mathcal{L} is provided to all baselines (including SAGE and GEAR) to ensure the comparison focuses on “memory and diagnosis” rather than “grasp pose learning”. 4.1.4 Experimental Tasks We designed three distinct tasks to evaluate the system’s performance across long-horizon planning, large-scale memory retention, and dynamic resilience. To ensure relevance to RCIM readers, these tasks are modeled after core scenarios in flexible manufacturing and intralogistics. Task 1: Long-Horizon Multi-Stage Assembly (Simulation) Modeled after a “PCB Component Kitting and Insertion” process. The robot operates in a multi-zone workcell ("Factories") containing various "components" (blocks/cylinders representing capacitors/chips). Goal: The VLM must parse a high-level manufacturing order (e.g., "Assemble the Red Kit and transfer to Zone C"), decomposing it into a sequence of 10+ manipulation steps (pick, place, stack), requiring strict adherence to geometric constraints and logical dependencies. Task 2: Large-Scale Facility Inventory (Simulation) Simulates an automated inventory management task in a large-scale warehouse or town environment. Goal: The robot performs a random-walk exploration to build a semantic map of the facility. The system is then subjected to a rigorous "Inventory Audit" (Q&A phase), assessing its ability to recall the location ("Where is the emergency stop?"), count ("How many pallets?"), and spatial relations of entities across the entire facility, testing the capacity of DEWM to handle large-scale static data. Task 3: Dynamic Exception Handling (Real-World) Replicates a “High-Mix Low-Volume Assembly Line” prone to runtime disturbances. Goal: The robot performs a standard pick-and-place assembly on the physical Franka platform. We introduce real-world anomalies common in factory settings, such as part slippage (gripper failure) and unexpected obstacles (safety stops). The system must diagnose these interruptions in real-time and generate "Informed Recovery" strategies (e.g., "Re-grasp" or "Clear Obstacle") to minimize downtime, rather than blindly halting or retrying. 4.1.5 Evaluation Metrics Let N be the total number of trials and \mathbb{I}(\cdot) be the indicator function. We employ five quantitative metrics to evaluate system performance: a. Industrial Reliability (Task Success Rate - TSR) Serves as the primary measure of end-to-end operational effectiveness (proxy for manufacturing yield rate). A trial is deemed successful only if all physical sub-goals are met and (for Task 2) all verification questions are answered correctly. TSR=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}(\text{Trial}_{i}=\text{Success})\times 100\% (13) where N is the total number of experimental trials. b. Digital Twin Fidelity (State Tracking Accuracy - STA) Quantifies the accuracy of the DEWM in maintaining object states when they are Out-of-View (OOV), acting as a metric for process monitoring quality. At each timestep t , we compare the estimated state S_{EWM}(o) against the simulation ground truth S_{GT}(o) , averaged over the task duration T : STA=\frac{1}{T}\sum_{t=1}^{T}\frac{\sum_{o\in\mathcal{O}_{t,OOV}}\mathbb{I}(S_{EWM}(o)=S_{GT}(o))}{|\mathcal{O}_{t,OOV}|}\times 100\% (14) This metric rigorously evaluates the "Object Permanence" capability required for reliable inventory tracking. c. Operational Cost & Latency (Planning Efficiency - PE) Acts as a proxy for economic cost (Token consumption) and cycle time (Inference latency). It is defined as the total count of VLM API calls per trial ( N_{calls} ), separating initial planning ( N_{init} ) from error-induced replanning ( N_{replan} ): PE=N_{calls}=N_{init}+N_{replan} (15) A lower PE indicates a stable memory context that minimizes expensive model queries, directly translating to a more cost-effective production cycle. d. Memory Storage Efficiency (Information Efficiency - IE) Measures the storage density of the system, critical for long-term deployment. It is calculated as the ratio of valid information queries answered ( N_{Info} ) to the total number of memory entries ( N_{Items} ) stored in the database: IE=\frac{N_{Info}}{N_{Items}}\times 100\% (16) High IE indicates that the system stores highly relevant, structured knowledge rather than redundant raw logs (as seen in baseline SAGE). e. Diagnostic Precision (Causal Diagnosis Accuracy - CDA) Evaluates the system resilience and ability to minimize “Mean Time to Recovery (MTTR)”. It measures the percentage of diagnostic attempts where the system-generated reason R_{gen} matches the ground truth reason R_{GT} : CDA=\frac{1}{M}\sum_{j=1}^{M}\mathbb{I}(R_{gen,j}=R_{GT,j})\times 100\% (17) where M is the total number of diagnostic attempts (not failure events), as a single failure may require multiple diagnosis rounds before successful recovery. Unlike “Blind Retry” strategies (CDA=0), a high CDA enables “Informed Replanning”, significantly reducing downtime. Ground Truth Labeling Protocol: To ensure objectivity and reproducibility, R_{GT} is determined through a post-hoc sensor log analysis protocol rather than subjective judgment. Specifically, for each induced failure event, we record synchronized multi-modal logs including: (1) gripper width and force-torque readings from the Franka controller, (2) RGB-D frames at 10 Hz, and (3) joint states and end-effector poses. 4.2 Main Results: Comparison with Baselines We conducted comprehensive comparisons against the baselines defined in Section 4.1.1 across both simulation and real-world platforms. The results for the three core tasks are presented in Tables 2 , 4 , and 5 . 4.2.1 Task 1: Long-Horizon Assembly & Reasoning (Simulation) Task 1 evaluates the system’s ability to handle multi-step physical interactions and maintain state persistence. The scene comprises three distinct zones (“Factories”) containing blocks, spheres, and cylinders (see Fig. 5 ). Figure 5 : Task 1 Environment Setup. The multi-zone factory simulation scene containing blocks, spheres, and cylinders distributed across three distinct platforms. This raw visual input triggers the memory construction process. a. Memory Construction Analysis Starting from the raw simulation input (Fig. 5 ), the system successfully decouples the static background \mathcal{S}_{bg} from dynamic entities \mathcal{S}_{obj} (Fig. 6 a-b). Simultaneously, the initial topological state is grounded in the Semantic Graph \mathcal{G} (Fig. 6 c). (a) Background ( \mathcal{S}_{bg} ) (b) Object Registry ( \mathcal{S}_{obj} ) (c) Initial Semantic Graph ( \mathcal{G} ) & Adjacency Matrix Figure 6 : Initial Memory Construction in Task 1. (a-b) The Geometric Layer of DEWM separates the static background (Left) from dynamic objects (Right). (c) The Semantic Layer initializes the topological graph and its matrix representation, anchoring the scene’s logical structure before manipulation. b. Reasoning and Execution Analysis The reasoning process is captured in Fig. 7 . Specifically, the robot received the high-level instruction: “Transport the red cube from Zone A and stack it onto the blue cylinder in Zone B.” The Task Memory \mathcal{M} (Fig. 7 a) decomposes the high-level instruction into a structural plan. Most importantly, by comparing the initial matrix in Fig. 6 (c) with the post-execution matrix in Fig. 7 (b), we observe the dynamic graph update: relations are rewritten in real-time upon action completion (e.g., On(cube, table) \to On(cube, cylinder) ). This capability explains our high STA scores compared to the static memory of GEAR. (a) VLM-Generated Task Topology Path ( \mathcal{M} ) (b) Post-Execution Dynamic Update Figure 7 : Reasoning and Dynamic Update Process. (a) Based on the initial memory, the VLM decomposes the instruction “Complete Red Assembly” into a hierarchical execution tree. (b) After execution, the DEWM reflects the new physical reality. By comparing this with Fig. 6(c) , we observe the real-time topology update (e.g., edge On(cube, table) is replaced by On(cube, cylinder) ), verifying the system’s dynamic tracking capability. c. Quantitative Analysis As shown in Table 2 , VLM-DEWM outperforms all baselines ( N=50 trials). 1. Ours: We achieved a 94.0% TSR with near-perfect 100% STA. The PE (75) is higher than pure VLMs because our architecture actively invokes the VLM for verification and geometric adjudication (Sec. 3.3.3 ), whereas pure VLMs rely on a single, hallucination-prone generation (PE=1). Crucially, our 100% IE confirms that the structured memory ( \mathcal{G},\mathcal{S} ) stores zero redundant information. 2. Memory-Enhanced Baselines: SAGE achieved a moderate 72% TSR but suffered from severe inefficiency (PE=261, IE=12.26%). Its unstructured “agent-to-agent” dialogue logging forces the VLM to parse redundant text, degrading performance. GEAR failed completely (0% TSR) because its static RAG knowledge base could not update object locations after manipulation, leading to fatal conflicts between memory and reality. 3. Pure VLMs: Despite receiving initial positions, pure VLMs struggled with the long horizon, exhibiting severe hallucination (TSR 0–28%). Only the strongest models (Gemini 2.5 Pro, GPT-5) managed sporadic successes. Table 2 : Quantitative Results for Task 1 (Long-Horizon Assembly) Method VLM Core TSR (%) \uparrow STA (%) \uparrow PE \downarrow IE (%) \uparrow VLM-DEWM (Ours) Gemini 2.5 Pro 94.00 100.00 75 100.00 SAGE [ 18 ] Gemini 2.5 Pro 72.00 83.33 261 12.26 GEAR [ 21 ] Gemini 2.5 Pro 0.00 – 148 – Gemini 2.5 Pro Gemini 2.5 Pro 28.00 39.74 1 – GPT-5 GPT-5 12.00 34.62 1 – Qwen3-VL Qwen3-VL 0.00 30.77 1 – GLM-4.5V GLM-4.5V 0.00 15.38 1 – Note: Pure VLMs achieve PE=1 due to a single long-context prompt, but fail to complete the task (Low TSR). 4.2.2 Task 2: Large-Scale Exploration (Simulation) This task evaluates memory construction and zero-shot spatial reasoning in a town-scale environment. We introduce “Query Success Rate (QSR)” as the primary metric, assessed via a graded question set (Table 3 ). Figure 8 : Task 2 Environment Setup. Multi-modal inputs (LIDAR + Camera) + semantic segmentation acquired during the town-scale exploration. This raw sensory stream serves as the input for constructing the DEWM. Table 3 : Task 2 Spatial Question Set Distribution Difficulty Count Description Easy 15 Basic Recognition (e.g., “Is there a bank?”) Medium 20 Spatial Relations (e.g., “Is the bench near the park?”) Hard 5 Multi-step Reasoning (e.g., Navigation heuristics) High 10 Daily Scenario Q&A a. Hierarchical Memory Construction The DEWM construction process is visualized in Fig. 9 . Starting from raw inputs, the system builds a persistent global map \mathcal{S}_{bg} and extracts key landmarks into the object registry \mathcal{S}_{obj} (Fig. 9 a-b). These geometric entities are abstracted into the town-scale semantic graph \mathcal{G} (Fig. 9 c), which serves as the knowledge base for answering the Q&A set defined in Table 3 . (a) Global Map ( \mathcal{S}_{bg} ) (b) Landmarks ( \mathcal{S}_{obj} ) (c) Semantic Graph Topology ( \mathcal{G} ) Figure 9 : Hierarchical Memory Construction in Task 2. (a) The reconstructed global semantic map serves as the metric substrate. (b) Key landmarks (e.g., hydrants) are extracted as object-centric instances. (c) The Semantic Graph abstracts these entities into topological relations (e.g., Near(bench, park) ), enabling efficient spatial querying. b. Exploration Process Analysis In this scenario, the robot performed a 240-step random-walk exploration mission to map the unknown facility. The temporal evolution of this mission is captured in the exploration history (Fig. 10 ). The sparsity of Green nodes (New Entity) versus Yellow nodes (Re-observation) in the tree visually demonstrates our high Information Efficiency (IE), as the system effectively recognizes previously seen areas without redundant memory writes, creating a compact yet comprehensive map. Figure 10 : Exploration Execution History ( \mathcal{M} ). The temporal tree visualizes the robot’s 240-step exploration path. Color-coded nodes indicate memory events: Green nodes mark steps where new entities were discovered and registered into \mathcal{G} ; Yellow nodes indicate steps with only re-observations, demonstrating the system’s efficiency in avoiding redundant updates. c. Quantitative Analysis As shown in Table 4 , VLM-DEWM exhibits overwhelming advantages in memory quality. 1. Ours: We achieved 94.0% QSR and 90.14% IE . By structuring the town into a hierarchical graph \mathcal{G} , our system converts spatial reasoning into efficient database queries rather than text processing. The PE (509) reflects the necessary cost of building a persistent map over 240 observation points. 2. Baselines: SAGE showed disastrous efficiency (PE=2162, IE=1.30%). Its unstructured logging forced the VLM to perform “needle-in-a-haystack” retrieval on redundant text, limiting QSR to 52%. Pure VLMs acted as streaming processors with no long-term retention (STA \approx 29%), failing to answer cross-regional queries. Table 4 : Quantitative Results for Task 2 (Exploration & Q&A) Method VLM Core QSR (%) \uparrow STA (%) \uparrow PE \downarrow IE (%) \uparrow VLM-DEWM (Ours) Gemini 2.5 Pro 94.00 92.68 509 90.14 SAGE Gemini 2.5 Pro 52.00 56.10 2162 1.30 GEAR Gemini 2.5 Pro 74.00 - 240 - Gemini 2.5 Pro Gemini 2.5 Pro - 29.27 240 - 4.2.3 Task 3: Real-World Dynamic Recovery Finally, we validated the system on the physical Franka platform (Fig. 11 ) using a dynamic assembly task with induced failures. Fairness Note: As specified in Section 4.1.3 , all methods utilized the same Shape Prior Library ( \mathcal{L} ) for grasp pose generation. This isolates the evaluation to strictly focus on memory, state tracking, and diagnosis , decoupling it from low-level manipulation skill proficiency. Figure 11 : Real-World Deployment. The Franka Emika Panda performing the dynamic assembly task. The system successfully diagnoses and recovers from induced execution failures using the CS -based causal diagnosis. a. Mechanism and Qualitative Analysis Figure 11 illustrates the mechanism behind this resilience. In a standard cycle, the robot is tasked to retrieve a gear and insert it into a shaft. During the transport phase, we induced a “Gripper Slip” failure, causing the part to fall onto the table. The core challenge in Task 3 is the dynamic evolution of the environment (e.g., new obstacles appearing during failure). Baselines like SAGE and GEAR rely on a static snapshot memory; once the environment changes, their knowledge base suffers from epistemic stagnation , leading to inevitable planning conflicts. In contrast, VLM-DEWM maintains a live topology. When the transport failure occurs (Fig. 11 , bottom), the system detects the topological breach (Expected: Holding , Actual: Idle ). Instead of hallucinating success or freezing, it autonomously triggers a “Re-perception and Search” routine, effectively closing the cognitive loop in the physical world. b. Quantitative Analysis Table 5 reveals a stark contrast: VLM-DEWM achieved an 95.0% TSR, while all baselines failed completely (0% TSR). 1. Cognitive Resilience via Iterative Diagnosis (Ours): A critical insight lies in the relationship between CDA (36.96%) and TSR (95.0%). Among the 20 induced failures, the system diagnosed the root cause immediately in only 6 cases. However, in 11 cases, it successfully anchored the failure after multiple diagnostic rounds ( N7.815 is rejected as a statistical anomaly or noise. B.6 Long-Term Memory Lifecycle Management To ensure robustness in long-horizon operations, the “Memory Curation” is not merely a deletion process but a full Entity Lifecycle Management system. We define three states for every memory entity: Active , Uncertain , and Archived . a. State Transitions and Decay Dynamics: When the robot performs a perception scan in Zone Z_{k} , the confidence c_{t} of an unobserved entity v_{i} follows an exponential decay model, representing the increasing probability that the object has been moved or removed: c_{t}(v_{i})=c_{t-1}(v_{i})\cdot\lambda_{decay} (41) Entities transition states based on thresholds: 1. Active \to Uncertain: When c_{t}<\tau_{uncertain} (set to 0.6). The planner adds a "Verify" sub-task before interacting. 2. Uncertain \to Archived: When c_{t}<\tau_{archive} (set to 0.1). The entity is removed from the spatial index tree (KD-Tree) to optimize query speed but retained in cold storage (Disk). b. Restoration of Archived Entities (The "Zombie" Problem): A critical feature is the ability to recover from false negatives. If a new observation o_{new} appears but fails to match any Active entity, the system queries the Archived database. If a match is found (using the logic in B.1), the entity is restored to Active state with c_{t} reset to 1.0, preserving its historical semantic attributes (e.g., "fragile"). c. Curation Algorithm: The complete lifecycle maintenance logic is detailed in Algorithm 3 . Algorithm 3 Entity Lifecycle Maintenance 1: DEWM \Omega , Current Zone Z_{k} , Observed IDs \mathcal{O}_{k} 2: \mathcal{K}_{k}\leftarrow\Omega.\text{GetEntitiesInZone}(Z_{k}) 3: \mathcal{U}_{k}\leftarrow\mathcal{K}_{k}\setminus\mathcal{O}_{k} \triangleright Unobserved entities 4: // Phase 1: Decay and Archive 5: for id in \mathcal{U}_{k} do 6: v\leftarrow\Omega.\text{GetEntity}(id) 7: v.confidence\leftarrow v.confidence\times\lambda_{decay} 8: if v.confidence<\tau_{archive} then 9: \Omega.\text{ArchiveEntity}(id) \triangleright Move to cold storage 10: else if v.confidence<\tau_{uncertain} then 11: v.state\leftarrow\texttt{Uncertain} 12: end if 13: end for 14: // Phase 2: Reinforcement and Restoration 15: for id in \mathcal{O}_{k} do 16: if id\in\Omega.\text{Archived} then 17: \Omega.\text{RestoreEntity}(id) \triangleright Restoration Event 18: end if 19: v\leftarrow\Omega.\text{GetEntity}(id) 20: v.confidence\leftarrow 1.0 21: v.state\leftarrow\texttt{Active} 22: end for Parameter Rationale: We select \lambda_{decay}=0.95 . This implies a "half-life" of approx 13 scans. The archiving threshold \tau_{archive}=0.1 requires \approx 45 consecutive misses, providing a robust buffer against transient occlusions common in collaborative construction. B.7 Force-Torque Feedback Integration While VLM-DEWM focuses on cognitive-level planning (not low-level impedance control), the system interfaces with the robot controller’s force-torque (F/T) signals to trigger state transitions in the CS module. This section specifies the integration protocol. a. Scope Clarification We do not design force control laws. Instead, we treat F/T readings as binary event triggers that signal the success or failure of contact-rich actions to the cognitive layer. The underlying impedance/admittance control is handled by the robot’s native controller (Franka Control Interface in our experiments). b. Threshold-Based Event Detection For each contact-rich action type, we define conservative thresholds based on the Franka Emika Panda’s specifications and empirical tuning: Table 8 : Force-Torque Thresholds for State Transition Triggers Action Signal Threshold Triggered Event Grasp Gripper Force F_{g} <5 N (after close) Grasp Failure Grasp Gripper Force F_{g} >40 N Over-squeeze (abort) Transport End-effector |F_{ext}| >15 N Collision Detected Place End-effector F_{z} >10 N (sustained) Contact Confirmed Insert End-effector |\tau_{ext}| >2 Nm Jamming Detected c. Mapping to CS State Transitions The F/T events are mapped to CS updates as follows: 1. Grasp Failure ( F_{g}<5 N after gripper close command): CS remains at \langle\texttt{Idle},\texttt{NULL}\rangle ; ERT’s \mathcal{A}_{causal} is invalidated. 2. Collision Detected ( |F_{ext}|>15 N during transport): Motion halts; CS transitions to \langle\texttt{Interrupted},\text{last\_target}\rangle ; triggers re-perception. 3. Contact Confirmed ( F_{z}>10 N sustained for 0.5s during place): CS transitions from \langle\texttt{Placing},\text{obj}\rangle to \langle\texttt{Idle},\texttt{NULL}\rangle ; placement success logged. d. Implementation Note These thresholds are fixed in our experiments, calibrated once for the Franka+Robotiq platform. For deployment on different hardware, re-calibration is required. Adaptive thresholding based on object properties (e.g., fragile objects use lower F_{g} limits) is a direction for future work (Section 4.4.2 ). Appendix C Implementation Details of Dynamic Context Engineering This appendix details the subgraph extraction algorithm and prompt synthesis logic for the context engineering pipeline described in Section 3.3.2 . C.1 Task-Relevant Subgraph Extraction Algorithm The “Information Bottleneck” principle in Section 3.3.2 is operationalized through a two-stage filtering algorithm, not learned compression. We prioritize interpretability and determinism over optimality. C.1.1 Stage 1: Zone-Based Spatial Filtering Given the current subtask ss_{t} and robot state CS_{t} , we first identify the relevant zones : \mathcal{Z}_{relevant}=\{Z_{current}\}\cup\{Z\in\mathcal{G}_{zone}\mid Z\text{ mentioned in }ss_{t}.\text{goal}\} (42) This exploits the hierarchical structure of \mathcal{G} to immediately prune objects in distant, task-irrelevant areas. Example: For ss_{t}= “Pick red_block from Zone_A”, we have \mathcal{Z}_{relevant}=\{\text{Zone\_A}\} , ignoring all objects in Zone_B, Zone_C, etc. C.1.2 Stage 2: Task-Driven Relevance Scoring Within \mathcal{Z}_{relevant} , we compute a relevance score for each vertex v_{i}\in\mathcal{G}_{obj} : \begin{split}r(v_{i})&=w_{1}\cdot\mathbb{I}[v_{i}\in ss_{t}.\text{args}]\\ &\quad+w_{2}\cdot\mathbb{I}[\exists e:(v_{i},v_{j})\land v_{j}\in ss_{t}.\text{args}]\\ &\quad+w_{3}\cdot\text{sim}(v_{i}.\text{cat},ss_{t}.\text{keywords})\end{split} (43) where: 1. w_{1}=1.0 : Direct task arguments (e.g., the object to be picked) 2. w_{2}=0.5 : 1-hop neighbors of task arguments (e.g., the surface it’s on) 3. w_{3}=0.2 : Semantic similarity to task keywords (e.g., “container” for a packing task) 4. \text{sim}(\cdot) : Embedding cosine similarity (precomputed, not runtime VLM call) C.1.3 Subgraph Construction The extracted subgraph G_{sub} is formed by selecting vertices that exceed a relevance threshold: \displaystyle V_{sub} \displaystyle=\{v_{i}\mid r(v_{i})>\tau_{rel}\}\cup\{v_{robot}\} (44) \displaystyle E_{sub} \displaystyle=\{e_{ij}\in\mathcal{E}\mid v_{i}\in V_{sub}\land v_{j}\in V_{sub}\} (45) We set \tau_{rel}=0.3 empirically. This value was chosen to effectively prune irrelevant background objects (typically r<0.2 ) while retaining indirect dependencies (e.g., support surfaces with r\approx 0.5 ). This filtering typically yields |V_{sub}|\approx 3 – 8 vertices for standard manipulation tasks. C.1.4 Algorithm Pseudocode Algorithm 4 Task-Relevant Subgraph Extraction 1: \mathcal{G} , ss_{t} , CS_{t} , threshold \tau_{rel} 2: G_{sub}=(V_{sub},E_{sub}) 3: \mathcal{Z}_{rel}\leftarrow\textsc{GetRelevantZones}(ss_{t},CS_{t}) 4: V_{candidates}\leftarrow\{v\in\mathcal{G}_{obj}\mid v.\text{zone}\in\mathcal{Z}_{rel}\} 5: V_{sub}\leftarrow\emptyset 6: for v_{i}\in V_{candidates} do 7: r_{i}\leftarrow\textsc{ComputeRelevance}(v_{i},ss_{t}) \triangleright Eq. above 8: if r_{i}>\tau_{rel} then 9: V_{sub}\leftarrow V_{sub}\cup\{v_{i}\} 10: end if 11: end for 12: E_{sub}\leftarrow\{(v_{i},v_{j})\in\mathcal{E}\mid v_{i},v_{j}\in V_{sub}\} 13: return (V_{sub},E_{sub}) C.1.5 Clarification on “Information Bottleneck” We use the Information Bottleneck (IB) principle as a conceptual motivation , not a learned objective. Our rule-based extraction approximates the IB goal of minimizing I(Z;\Omega) (irrelevant info) while maximizing I(Z;Y) (task-relevant info) through: 1. Zone filtering \approx spatial compression 2. Relevance scoring \approx task-conditional selection A learned IB encoder (e.g., via variational methods) is left for future work. C.2 Initialization and Task Decomposition Context Objective: To decompose a high-level user command into a structured Directed Acyclic Graph (DAG) of sub-tasks. a. Data Extraction: The system scans the initial environment \Omega_{0} and compiles a list of available entities: \mathcal{L}_{res}=\{(id_{i},cat_{i},\mathbf{p}_{i})\} . b. Prompt Template: Prompt: Task Decomposition Role: Expert Robotic Planner. User Command: "{T}" Available Resources: - {resource_list} Objective: Decompose the command into a sequence of sub-tasks. Output a JSON object representing a DAG, where each node contains: "id", "function" (e.g., pick, place), "args", and "dependencies". Example Output: { "nodes": [ { "id": 1, "function": "pick", "args": ["blue_block_1"], "dependencies": [] } ] } C.3 Coarse Motion Planning Context Objective: To generate collision-free waypoints for macro-navigation. a. Data Extraction: We cluster the background map \mathcal{S}_{bg} (e.g., using DBSCAN) to extract obstacle bounding boxes \mathcal{O}_{list} . b. Prompt Template: Prompt: Coarse Path Planning Current Goal: "{SS_t.description}" Robot State: {CS_t} Constraints: - Obstacles (BBoxes): {obstacle_list} - Semantic Constraints: {relevant_relations_from_G} Task: Generate a list of SE(3) waypoints for the end-effector to reach the target area while avoiding obstacles. C.4 Fine-Grained Manipulation Context Objective: To generate precise delta-pose adjustments for contact-rich interaction. a. Data Extraction: We query high-fidelity data sources: Shape Prior \mathcal{L} , Force-Torque sensors F_{t} , and precise object pose \mathbf{T}_{obj} . b. Prompt Template: Prompt: Fine Manipulation Target Object: {target.id} ({target.category}) - Pose (SE3): {target.pose} - Envelope ( \mu,\Sigma ): {target.envelope} - Shape Prior: {shape_desc} - Force Readings: {force_torque_data} Task: Based on the geometry and force feedback, compute the next optimal delta-pose ( \Delta\mathbf{T} ) for the end-effector. Return a JSON with "action_type" and "value". C.5 Failure Diagnosis and Recovery Context Objective: To analyze execution failures and propose a recovery plan. a. Data Extraction: We compile the Action Sequence History \mathcal{H}_{ash} and the post-failure world state \Omega_{fail} . b. Prompt Template: Prompt: Failure Diagnosis CRITICAL FAILURE REPORT 1. Failed Action: "{failed_SS_t.description}" 2. Context: - State Before: {CS_t_before} - Action Log: {recent_action_log} 3. Post-Mortem: - Error Code: {failure_reason} - Current World State: {summary_of_S_and_G} Task: 1. Diagnose the root cause of the failure. 2. Propose a new TTP (DAG JSON) to recover and achieve the original goal. C.6 Formal Definition of Semantic Discrepancy Operator ( \ominus ) The semantic discrepancy \Delta=\Omega_{t+1}\ominus\mathcal{A}_{causal} in Section 3.3.5 is computed via a rule-based state comparison , not probabilistic VLM inference. This ensures deterministic and auditable diagnosis. a. Definition Given the expected post-state \mathcal{A}_{causal}=\langle CS_{exp},\mathcal{R}_{exp}\rangle (where CS_{exp} is the expected constraint state and \mathcal{R}_{exp} is a set of expected relation changes) and the actual observed state \Omega_{t+1} , we define: \Delta=\Omega_{t+1}\ominus\mathcal{A}_{causal}:=\langle\Delta_{CS},\Delta_{\mathcal{G}},\Delta_{\mathcal{S}}\rangle (46) b. Component-wise Computation: b.1. Constraint State Discrepancy ( \Delta_{CS} ): A direct symbolic comparison between expected and actual micro-topological states: \Delta_{CS}=\begin{cases}\texttt{NULL}&\text{if }CS_{t+1}=CS_{exp}\\ (CS_{exp},CS_{t+1})&\text{otherwise}\end{cases} (47) Example: If CS_{exp}=\langle\texttt{Holding},\text{obj\_1}\rangle but CS_{t+1}=\langle\texttt{Idle},\texttt{NULL}\rangle , then \Delta_{CS}=(\texttt{Holding}\to\texttt{Idle}) , indicating a grasp failure. b.2. Semantic Graph Discrepancy ( \Delta_{\mathcal{G}} ): We perform a structural graph difference on the object-level semantic graph: \displaystyle\Delta_{\mathcal{G}}^{+} \displaystyle=\mathcal{E}_{t+1}\setminus\mathcal{E}_{exp}\quad\text{(unexpected new edges)} (48) \displaystyle\Delta_{\mathcal{G}}^{-} \displaystyle=\mathcal{E}_{exp}\setminus\mathcal{E}_{t+1}\quad\text{(missing expected edges)} (49) where \mathcal{E} denotes the edge set of \mathcal{G}_{obj} . This is implemented as a deterministic set operation on relation tuples. Example: If the action was Place(A, B) with \mathcal{R}_{exp}=\{+\texttt{On}(A,B)\} , but post-observation shows \texttt{On}(A,\texttt{floor}) , then: 1. \Delta_{\mathcal{G}}^{-}=\{\texttt{On}(A,B)\} (expected edge missing) 2. \Delta_{\mathcal{G}}^{+}=\{\texttt{On}(A,\texttt{floor})\} (unexpected edge present) b.3. Geometric Discrepancy ( \Delta_{\mathcal{S}} ): For objects involved in the action, we compute the pose deviation: \Delta_{\mathcal{S}}(o_{i})=\|\boldsymbol{\mu}_{t+1}(o_{i})-\boldsymbol{\mu}_{exp}(o_{i})\|_{2} (50) A significant deviation ( >\tau_{geo\_drift} , default 0.05m) flags potential unmodeled disturbances. c. Discrepancy Report Generation: The structured \Delta is serialized into natural language for VLM-based recovery planning: Discrepancy Report Template DISCREPANCY DETECTED - CS Mismatch: Expected { CS_{exp} }, Actual { CS_{t+1} } - Missing Relations: { \Delta_{\mathcal{G}}^{-} } - Unexpected Relations: { \Delta_{\mathcal{G}}^{+} } - Geometric Drift: {object: drift_distance} Likely Cause: [Inferred from \Delta_{CS} pattern, see Table 9 ] d. Diagnosis Rule Table: The \Delta_{CS} pattern deterministically maps to failure categories: Table 9 : Rule-Based Failure Diagnosis from \Delta_{CS} CS_{exp} CS_{t+1} Diagnosed Failure Holding(X) Idle Grasp Slip / Gripper Failure Holding(X) Holding(Y) Object Misidentification Approaching(X) Idle Motion Interrupted / Collision Placing(X, Y) Holding(X) Placement Rejected / Unstable This rule-based approach ensures that the \ominus operator produces deterministic, explainable discrepancy reports, while the subsequent VLM call ( C.5 ) is responsible only for recovery strategy synthesis —not diagnosis itself. Appendix D Implementation Details of the ERT Validation Protocol This appendix details the schema and verification logic for the External Reasoning Trace (ERT) described in Section 3.3.3 . D.1 ERT Schema Definition The VLM output must strictly adhere to the schema defined in Table 10 . Table 10 : Schema Definition for External Reasoning Trace (ERT) Field Type Description & Example action_proposal String The proposed skill call. Ex: "Grasp(blue_block_1)" world_belief Object Structured assertion about state. Ex: { "pred": "On", "args": ["A", "B"] } causal_assump String Expected outcome. Ex: "CS becomes Holding(A)" confidence Float Self-evaluated score [0.0, 1.0]. fallback String Contingency plan. D.2 Multi-Layered Validation Logic The GetValidERT() function enforces a three-stage check: a. Syntactic Validation: We use a JSON schema validator to ensure the output contains all required fields (Table 10 ) and that data types are correct (e.g., 0.0\leq\texttt{confidence}\leq 1.0 ). b. Semantic Validation: We extract all entity IDs from action_proposal and world_belief . The validator queries the DEWM to ensure existence: \forall id\in\mathcal{E}_{ert},\quad(id\in\mathcal{G}_{obj})\lor(id\in\mathcal{G}_{zone}) (51) This prevents hallucinations of non-existent objects. c. Physical Consistency Validation: We verify the world_belief against the geometric ground truth \mathcal{S} . 1. Verify On(A, B): (a) Support: Check if A ’s centroid \boldsymbol{\mu}_{A} projected onto the XY plane lies within B ’s XY-envelope. (b) Contact: Check if the vertical gap is minimal: |Z_{min}(A)-Z_{max}(B)|<\epsilon_{contact} . 2. Verify Inside(A, B): Check if A ’s bounding box is fully contained within B ’s boundaries: (\mathbf{min}_{A}\geq\mathbf{min}_{B})\land(\mathbf{max}_{A}\leq\mathbf{max}_{B}) (52) 3. Verify Near(A, B): Check Euclidean distance: \|\boldsymbol{\mu}_{A}-\boldsymbol{\mu}_{B}\|