Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoning

Jooyoung Kim, Wonje Choi, Younguk Song, Honguk Woo · 2026

Read Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoning by Jooyoung Kim, Wonje Choi, Younguk Song, Honguk Woo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18495.txt.

Introduction

Kim and co-authors present NeSyCR, a framework that adapts robot code from video demos to new settings. Sight and physics gaps between demo and real use break direct code reuse. Demos first become symbolic paths that hold the core steps of the task. What-if states then show where the new setting clashes with those steps. A checked search over the state space writes fixed code, and task success rose 31.14% over Statler.

Catalog record

CREATORJooyoung Kim, Wonje Choi, Younguk Song, Honguk Woo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE106377
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18495

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoning Jooyoung Kim, Wonje Choi, Younguk Song, Honguk Woo Abstract Recent advances in Vision-Language Models (VLMs) have enabled video-instructed robotic programming, allowing agents to interpret video demonstrations and generate executable control code. We formulate video-instructed robotic programming as a cross-domain adaptation problem, where perceptual and physical differences between demonstration and deployment induce procedural mismatches. However, current VLMs lack the procedural understanding needed to reformulate causal dependencies and achieve task-compatible behavior under such domain shifts. We introduce NeSyCR , a neurosymbolic counterfactual reasoning framework that enables verifiable adaptation of task procedures, providing a reliable synthesis of code policies. NeSyCR abstracts video demonstrations into symbolic trajectories that capture the underlying task procedure. Given deployment observations, it derives counterfactual states that reveal cross-domain incompatibilities. By exploring the symbolic state space with verifiable checks, NeSyCR proposes procedural revisions that restore compatibility with the demonstrated procedure. NeSyCR achieves a 31.14% improvement in task success over the strongest baseline Statler, showing robust cross-domain adaptation across both simulated and real-world manipulation tasks. 1 Introduction Advances in foundation models have accelerated progress toward general-purpose embodied intelligence, enabling robots to interpret human instructions and execute complex tasks as autonomous control policies 15 ; 7 ; 26 ; 51 . In particular, Large Language Models (LLMs) with code-writing capabilities have inspired the Code-as-Policies paradigm, in which executable control code is synthesized from language instructions using predefined APIs ( 36 ; 54 ; 24 ; 25 ) . Furthermore, Vision-Language Models (VLMs) have extended this paradigm toward a general form of video-instructed robotic programming, where robotic programs are generated from instructional video demonstrations by translating observed task sequences into structured task specifications that can be compiled into control code 59 ; 61 ; 56 ; 65 . By capturing richer perceptual context and task intent from demonstrations, these methods enable more grounded robotic programming than text instructions alone. Figure 1 : Overview of NeSyCR in a drawer-organizing task scenario. (Left) Illustration of the domain gap between the demonstration and deployment. (Middle) Overview of NeSyCR framework, which generates an adapted procedure via neurosymbolic counterfactual reasoning. (Right) Outcome of the adapted procedure, showing that NeSyCR successfully executes the task via a grounded code policy. In video-instructed robotic programming, domain gaps between the demonstration and deployment are inevitable due to inherent differences in environmental layouts, object properties, and morphological constraints 45 ; 46 ; 16 ; 12 . While sensory observations can reveal physical discrepancies between the demonstration and the deployment environment, these observations alone do not explain how structural differences disrupt the underlying task procedure or how actions will causally unfold under altered conditions. To address this limitation, we present NeSyCR , a neurosymbolic framework that formulates cross-domain adaptation from demonstrations as counterfactual reasoning. At its core, NeSyCR infers how task outcomes would change under altered domain factors and proposes alternative actions that revise the demonstrated behavior to restore task-oriented behavioral compatibility. Given a demonstration, NeSyCR constructs a symbolic world model that encodes the task procedure as a symbolic abstraction of the trajectory. Leveraging this model, NeSyCR performs neurosymbolic counterfactual adaptation to revise the procedure for the deployment domain, enabling verifiable adaptation of the demonstrated procedure across domains. As shown in Figure 1 , the demonstration domain involves a human performing a drawer-organizing task with objects such as a magnetic hook and a box of screws placed on the table. In the deployment domain, however, the human hand is replaced by a robotic gripper, the magnetic hook is already arranged inside the drawer, and the screws are scattered across the surface. These differences create a procedural incompatibility: direct grasping becomes infeasible, requiring the magnetic hook to be repurposed as an auxiliary tool for gathering the screws. Furthermore, this modification introduces a cascading incompatibility: objects to be organized later in the procedure can obstruct the magnet, rendering it inaccessible. Resolving these coupled incompatibilities requires reordering the procedure so that the magnet is retrieved and used to aggregate the screws before the target objects are placed. However, a direct demo-to-code approach using VLM-based prompting would still attempt to grasp the scattered screws with the gripper, which fails due to their small size and dispersion. In contrast, NeSyCR identifies these incompatibilities through VLM-based symbolic translation and symbolic forward simulation. By combining the VLM’s commonsense reasoning with symbolic verification, it produces an adapted and coherent procedure that completes the task. Further details are illustrated in Figure 5 . To construct the symbolic world model, the VLM abstracts the demonstration into a symbolic trajectory that captures objects and their spatial and temporal relations, while the symbolic tool verifies these representations for logical consistency. In a deployment domain, NeSyCR identifies structural variations through this symbolic world model, detecting where the demonstrated procedure fails to reach the goal state under counterfactual states derived from target-domain observations. The VLM then proposes alternative action operators for the incompatible steps, and the symbolic tool verifies their causal validity, yielding revised procedural steps that restore compatibility. This reasoning process produces an adapted task specification that remains logically valid while preserving the task intent of the demonstration. The adapted specification is then compiled into a reliable, deployment-grounded code policy. We evaluate NeSyCR on a diverse set of video-instructed robotic programming scenarios, deploying robotic agents in both simulated and real-world environments. The cross-domain setting between demonstration and deployment is characterized by domain factors that capture variations in environmental and embodiment configurations. The scenarios consist of long-horizon manipulations involving multiple subtask types and requiring up to 116 visual-motor API calls, yielding compositional and procedurally complex tasks. NeSyCR outperforms the strongest baseline, Statler 66 , achieving an average improvement of 31.14% in task success rate, as reported in Tables 1 and 2 . Our contributions are summarized as follows: (1) We present the NeSyCR framework that casts cross-domain adaptation from demonstrations as counterfactual reasoning for video-instructed robotic programming. (2) We implement a VLM–symbolic tool pipeline that proposes and verifies alternative action steps, ensuring procedural compatibility across domains. (3) We evaluate NeSyCR across simulated and real-world robotic manipulation tasks using an experimental design that provides high granularity over domain factors and task complexity, enabling precise analysis of its cross-domain procedural adaptation. 2 Problem formulation We formulate the embodied domain as a tuple \mathcal{M}\!=\!(\mathcal{S},\!\mathcal{A},\!\mathcal{T},\!g) , where \mathcal{S} denotes the state space, \mathcal{A} the action space, \mathcal{T}\!:\!\mathcal{S}\!\times\!\mathcal{A}\!\rightarrow\!\mathcal{S} the transition function, and g\!\in\!\mathcal{G} the goal state. Under partial observability 52 , the agent receives observations o_{t}\!\in\!\mathcal{O} at each timestep t via an observation function \Omega\!:\!\mathcal{S}\!\times\!\mathcal{A}\!\rightarrow\!\mathcal{O} , which maps the underlying state to perceptual observations (e.g., RGB images). In cross-domain settings, a demonstration \mathcal{D}\!=\!(\!\{o_{t}\}_{t=1}^{N},\ell) specifies a domain \mathcal{M}_{\mathrm{S}}\!=\!(\mathcal{S}_{\mathrm{S}},\!\mathcal{A}_{\mathrm{S}},\!\mathcal{T}_{\mathrm{S}},\!g) performed under specific environmental and embodiment configurations, optionally with a language description \ell . The agent must achieve the same goal in a deployment domain \mathcal{M}_{\mathrm{T}}\!=\!(\mathcal{S}_{\mathrm{T}},\!\mathcal{A}_{\mathrm{T}},\!\mathcal{T}_{\mathrm{T}},\!g) , where \mathcal{S}_{\mathrm{S}}\!\neq\!\mathcal{S}_{\mathrm{T}} , \mathcal{A}_{\mathrm{S}}\!\neq\!\mathcal{A}_{\mathrm{T}} , or \mathcal{T}_{\mathrm{S}}\!\neq\!\mathcal{T}_{\mathrm{T}} . Our objective is to optimize a policy \pi_{\theta} from a single demonstration to maximize task success in the deployment domain: \pi_{\theta}^{*}=\argmax_{\pi_{\theta}}\mathbb{E}_{\tau\sim p(\cdot\mid\pi_{\theta},\mathcal{M}_{\mathrm{T}})}\big[\mathrm{SR}(\tau,g)-\lambda\mathrm{D}(\tau,\mathcal{D})\big] (1) where \tau\!=\!\{o_{1},a_{1},\dots,o_{N},a_{N}\} denotes a trajectory sampled from p(\tau\!\mid\!\pi_{\theta},\mathcal{M}_{\mathrm{T}}) , induced by executing \pi_{\theta} in \mathcal{M}_{\mathrm{T}} . Here, \mathrm{SR}(\tau,g) measures task success, \mathrm{D}(\tau,\mathcal{D}) measures the deviation between \tau and \mathcal{D} , and \lambda is a weighting factor. Thus, the policy \pi_{\theta}^{*} aims to

Read the complete text