Digital Twin - Guided Robot Path Planning: A Beta–Bernoulli Fusion with Large Language Model as a Sensor

Mani Amani, Reza Akhavian · 2025

Read Digital Twin - Guided Robot Path Planning: A Beta–Bernoulli Fusion with Large Language Model as a Sensor by Mani Amani, Reza Akhavian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20709.txt.

Introduction

Construction robots need site plans that reflect real hazards, not just design drawings. Mani Amani and Reza Akhavian fuse language guidance with building models in a 2025 paper on robot path planning. They treat a large language model as a sensor that scores danger.

Each obstacle in the building information model starts with a design-time repulsion value, modeled as a Beta distribution. The language model reads text descriptions and returns danger scores. Those scores update the distribution like new observations, a Beta-Bernoulli fusion. The updated values feed a potential-field planner that steers the robot. The result is a path cost that responds to context and wording, not only to plain distance.

Catalog record

CREATORMani Amani, Reza Akhavian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-20709

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Digital Twin - Guided Robot Path Planning: A Beta–Bernoulli Fusion with Large Language Model as a Sensor Mani Amani, Reza Akhavian Abstract Integrating natural language (NL) prompts into robotic mission planning has attracted significant interest in recent years. In the construction domain, Building Information Models (BIM) encapsulate rich NL descriptions of the environment. We present a novel framework that fuses NL directives with BIM–derived semantic maps via a Beta–Bernoulli Bayesian fusion by interpreting the LLM as a sensor: each obstacle’s design-time repulsive coefficient is treated as a Beta ⁡ ( α , β ) \mathrm{Beta}(\alpha,\beta) random variable and LLM-returned “danger scores” are incorporated as pseudo-counts to update α \alpha and β \beta . The resulting posterior mean yields a continuous, context-aware repulsive gain that augments a Euclidean-distance-based potential field for cost heuristics. By adjusting gains based on sentiment and context inferred from user prompts, our method guides robots along safer, more context-aware paths. This provides a numerically stable method that can chain multiple natural commands and prompts from construction workers and foreman to enable planning while giving flexibility to be integrated in any learned or classical AI framework. Simulation results demonstrate that this Beta–Bernoulli fusion yields both qualitative and quantitative improvements in path robustness and validity. Our code is publicly available on GitHub. \NameTag Amani, August 10, 2026 Figure 1 : Overview of the proposed BIM–LLM Bayesian fusion framework for danger-aware robotic mission planning. 1 Introduction The emergence of large language models (LLMs) has fundamentally transformed natural language processing (NLP) capabilities across diverse domains, with robotic planning experiencing similar paradigm shifts through enhanced semantic understanding and contextual reasoning [ 7 ] [ 12 ] . While there has been a concentrated focus on the advancement of prompted robotic applications, most of the research on language-commanded robotics is limited to predefined objects and primitives [ 3 ] . This restriction confines robots to a limited set of predefined actions. Sometimes, mission parameters must be adjusted and tuned given unseen and undefined prompts. Furthermore, it is well established that humans expect their collaborators, including AI, to interpret implicit cues beyond explicit wording [ 17 ] . This creates a need for a more flexible framework to analyze implicit sentiment information from natural language prompts. Our proposed BIM–LLM Bayesian fusion framework, illustrated in Figure 1 , addresses this gap. Since implicit and semantic information inherently depend on subjective human interpretation, a probabilistic framework provides a principled approach to quantify and propagate these uncertainties through the planning system. Specifically, the inherent ambiguity and context-dependency of natural language sentiment, combined with the possibility of LLM hallucinations, require a stable probabilistic approach. Therefore, leveraging deep learning models is considered a suitable approach since they have been shown to have high-accuracy sentiment analysis [ 4 ] . LLMs have demonstrated extensive success in sentiment analysis, positioning them as attractive candidates for assessing Theory of Mind-like associations [ 28 ] . Theory of Mind refers to the ability to attribute mental states (like beliefs, desires, and intentions) to oneself and others. In this context, the ’Theory of Mind-like associations’ means that the system is designed to infer and relate implicit emotional or cognitive states from verbal communication and, more generally, observed behavior [ 21 ] . While these advancements show promise in human-machine interaction, in robotics, the analysis must operate under strict mission constraints and heuristics, requiring that the sentiment data be transformed into actionable insights that can be adapted across various scenarios. This introduces additional complexity in transforming raw sentiment data into an actionable and generalizable format across diverse mission scenarios. Furthermore, for practical applications, it is essential to establish a general mapping between sentiment cues and environmental context to ensure accurate robot operations. Semantic maps have been extensively studied as a means to integrate qualitative and spatial information [ 15 ] . Typically, these maps assign a class label and descriptive textual information to spatial objects, enabling real-time map generation and planning. In many applied settings, there are existing data formats analogous to semantic maps. For example, BIM is a widely used format that represents both the three-dimensional geometry and associated textual information of a built environment [ 6 ] . Given its dual representation of spatial and semantic data, BIM serves as an excellent analog to a semantic map. Indeed, previous research has successfully leveraged BIM to provide the semantic information for robot planning [ 19 ] . Furthermore, the integration of internet of things (IoT) sensors, real-time data streams, and machine learning has driven an evolution from static BIM representations toward dynamic Digital Twins that continuously update to reflect current site conditions, enabling more responsive robotic navigation in changing construction environments. Previous works typically treat the LLM as the primary planning module, leveraging environmental information to generate semantically meaningful plans [ 25 ] . In contrast, we propose an alternative interpretation: using the LLM as a sensor rather than a planner. In this framework, the LLM provides semantic information that can be embedded into the map or any intermediate representation used during planning. The planning module itself remains agnostic to the LLM, allowing for the use of any classical or learning-based planner in conjunction with the LLM-as-sensor approach. Although visual–language maps have shown great promise [ 13 ] , most prior work focuses on robot planning from direct commands. In contrast, our contribution demonstrates that LLMs, combined with probabilistic map updates, can account for contextual and implicit sentiments that are not explicitly expressed in the natural-language command. Using NL as a cost signal for obstacle avoidance has been explored previously [ 18 ] . Prior work shows that VLMs can identify dynamic obstacles and assign danger scores. We extend this line of work by incorporating implicit and contextual information about the environment and by enabling the chaining and consolidation of multiple commands, yielding more flexible and general language-guided costmaps. Semantic planning has also been studied in the realm of visual-language-action models (VLA). However, these models are often expensive to train and deploy. Furthermore, these models often do not have formal verification and guarantees regarding their performance [ 20 ] . By incorporating NL data as sensor information that can augment costs, we can integrate NL with classical and certifiable AI algorithms such as A* for path-finding, which is commonly used in robotics [ 10 ] [ 7 ] . In the presented framework, the ideas proposed in studies such as [ 7 ] are extended to risk shaping rather than task satisfaction, with BIM - grounded semantics. Our contributions in this paper can be summarized as: 1. We propose a novel interpretation of LLMs as semantic sensors for robotic navigation, where natural language prompts about construction site conditions update obstacle danger coefficients through Beta-Bernoulli fusion. This approach leverages BIM family semantics to ground language understanding while maintaining map structure invariance. 2. We derive a closed-form Beta–Bernoulli fusion of prompt-derived danger scores into continuous repulsive gains, thereby formulating object avoidance as a repulsive potential-field cost metric. 3. We demonstrate that off-the-shelf LLMs correctly interpret domain-specific construction prompts (no domain fine-tuning needed), and that our fusion preserves the bounded-suboptimal guarantees of Multi-Heuristic A*. 2 Related Works Robotic policy learning aided by LLMs has been previously explored.

Read the complete text