Align While Search: Belief-Guided Exploratory Inference for World-Grounded Embodied Agents

Seohui Bae, Jeonghye Kim, Youngchul Sung, Woohyung Lim, [0.3em] LG AI Research · 2025

Read Align While Search: Belief-Guided Exploratory Inference for World-Grounded Embodied Agents by Seohui Bae, Jeonghye Kim, Youngchul Sung, Woohyung Lim, [0.3em] LG AI Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24461.txt.

Introduction

Seohui Bae, Jeonghye Kim, Youngchul Sung, and Woohyung Lim build an exploring agent for partly hidden worlds. Their agent is a language model system that adapts at test time. No gradient updates or extra training are allowed. The world is only partly observable. The agent must still act well inside it.

An external structured belief tracks the environment's state. Each action brings an observation that refines the belief. Actions are chosen to maximize predicted information gain over that belief. A light language-model surrogate estimates the gain. A new reward measures world alignment. It scores how well the posterior belief matches the true configuration. Experiments beat inference-time scaling baselines. Prompt-augmented and retrieval-enhanced language models both lost. The winning method also carried lower integration overhead.

Catalog record

CREATORSeohui Bae, Jeonghye Kim, Youngchul Sung, Woohyung Lim, [0.3em] LG AI Research
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-24461

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Align While Search: Belief-Guided Exploratory Inference for World-Grounded Embodied Agents Seohui Bae, Jeonghye Kim, Youngchul Sung, Woohyung Lim, [0.3em] LG AI Research Abstract In this paper, we propose a test-time adaptive agent that performs exploratory inference through posterior-guided belief refinement without relying on gradient-based updates or additional training for LLM agent operating under partial observability. Our agent maintains an external structured belief over the environment state, iteratively updates it via action-conditioned observations, and selects actions by maximizing predicted information gain over the belief space. We estimate information gain using a lightweight LLM-based surrogate and assess world alignment through a novel reward that quantifies the consistency between posterior belief and ground-truth environment configuration. Experiments show that our method outperforms inference-time scaling baselines such as prompt-augmented or retrieval-enhanced LLMs, in aligning with latent world states with significantly lower integration overhead. 1 Introduction Agents operating in partially-observable environments continuously encounter incomplete information in the course of accomplishing their goals ( 16 ; 14 ) . The key capability required in such settings is exploratory decision making: The agent should not only act to achieve the objective but also to collect information that refines its belief about the world. The interaction between action, observation, and belief refinement forms the basis of effective behavior under uncertainty. Figure 1 : Success rate vs. token usage on ALFWorld subtasks . AWS achieves higher success with 2-5 \times fewer tokens compared to strong inference-time baselines, highlighting efficient belief-guided exploration under partial observability. Full results in Appendix 20 . In this paper, we consider such exploratory decision making under uncertainty with focus on search task, which is the basic task for more advanced tasks, and examine inference-time world understanding under partial observability. While large language models (LLMs) have shown promise in zero-shot task execution ( 2 ; 25 ) , their static reasoning often fails to adapt to the unfolding dynamics of environments with partial observability. Prior approaches to such partial observability include train-time policy optimization via supervised or reinforcement learning training ( 26 ; 29 ; 42 ; 8 ) and inference-time scaling methods ( 22 ; 21 ; 30 ) . However, the former requires substantial training costs and limits deployment flexibility, while the latter lacks adaptive interaction with the environment. More recent inference-time agents, such as Reflexion ( 33 ) , RAP ( 10 ) , RAFA ( 24 ) , LAC ( 6 ) , and ReflAct ( 17 ) , couple LLMs with classical MDP planning or question-based belief summaries, but typically rely on additional simulators or learned critics and do not maintain an explicit probability distribution over latent environment configurations and object locations. We instead view exploratory decision making for search under partial observability as an approximate Bayes-adaptive control problem: the agent maintains a posterior over latent environment configurations and selects actions that trade off task reward and information gain in belief space. With this perspective, we propose a novel, lightweight but effective agent architecture that performs inference-time exploratory reasoning through posterior-guided belief refinement . Our core idea is as follows: the agent maintains a structured posterior belief over search action space, updates it based on environmental feedback, and selects actions that reflect this evolving belief. Thus, our behavior policy is driven not by memorized patterns but by a dynamically refined understanding of the world. Crucially, all our adaptation occurs at test time without gradient updates, fine-tuning, or additional environment models. Empirical results show that our inference-time alignment strategy outperforms inference-time policy optimization baselines with substantially lower computational overhead, as shown in Fig 1 . Our method generalizes across diverse object types, environments, and interaction histories without requiring task-specific training or reward-based tuning. Our contributions are summarized below: \bullet We introduce Align While Search (AWS), a lightweight world-aligned agent that casts test-time control as belief-guided search over object-centric hypotheses, without any additional training or gradient updates. \bullet We provide a Bayesian account of AWS by viewing it as approximate Bayes-adaptive control: a latent generative model with amortized posterior belief refinement implemented by an LLM, and information-gain-driven action selection on the belief state. \bullet Across ALFWorld, VirtualHome, and BabyAI, in both text-only and image-augmented settings, we show that AWS consistently improves search success–cost trade-offs over inference-time prompt/RAG scaling and train-time world-model or policy-gradient baselines, as seen in Fig. 1 . 2 Preliminaries MDP/POMDP for LLM agents. Many language-agent settings can be cast as a Markov Decision Process (MDP) \langle\mathcal{S},\mathcal{A},T,\mathcal{R},\gamma\rangle . In embodied environments such as ALFWorld and VirtualHome, the (latent) world state s includes agent pose, object locations, container open/close states, and inventory. The agent issues high-level action a ( e.g. , navigation, opening, manipulation) and receives textual observations o describing the partially observed scene. Then, the world state s changes according to transition T based on the current state and agent action. For an LLM-based agent, the input is typically the interaction history ( 42 ) , h_{t}=\{(o_{i},a_{i})\}_{i=1}^{t} , or a textual/symbolic summary ( 33 ) , and the LLM implements a policy \pi(a_{t}\mid h_{t}) . Concrete instantiations for ALFWorld/VirtualHome are given in Appendix 9.2 . Epistemic uncertainty and Bayes-adaptive MDPs. Because the environment is only partially observed, LLM agents in ALFWorld and VirtualHome are naturally modeled as POMDPs ( 16 ) : at each step the agent receives partial observation o\sim\mathcal{O}(o\mid s,a) about the environment state s . Beyond perceptual uncertainty, there is also epistemic uncertainty about latent environment parameters, such as object layouts or stochastic dynamics, which must be inferred from interaction. Bayes-adaptive MDPs (BAMDPs) ( 45 ; 15 ) make this explicit by introducing latent parameters \phi\in\Phi with prior b_{0}(\phi) and treating the belief over \phi as part of the state. Thus, in a BAMDP \langle\mathcal{S},\mathcal{A},\Phi,T,\mathcal{R},\gamma,b_{0}\rangle , the agent seeks a policy that maximizes expected return with posterior updates over \phi induced by the history, thereby motivating information-seeking behavior. 3 Pitfalls and Opportunities in LLM Agents Modern language agents, both base models and those post-trained with supervised trajectories, tend to overfit to trained behaviors. In this section, we investigate this phenomenon in partially observed settings. 3.1 Failure modes in search tasks We conducted an experiment to investigate the diversity and test-time adaptation of generated action sequences of an agent. In each run of this experiment, the agent needs to search for an object in a household, yielding one action sequence, and each run has a different room layout and a target location. The result is shown in Figure 2 (left), where the entropy represents the entropy of empirical distribution on the set of actions computed from all experiment runs, and the unique ratio represents the number of distinct action sequences out of total action sequences. It is seen that the base model (GPT-4o-mini) exhibits low action entropy (1.94) and low distinct trajectory ratio (0.21) compared to our method 3.11 and 0.5, respectively, revealing that the base LLM agent yields repetitive search behavior with low diversity in action, even though the environment changes run by run. More importantly, we observed that this behavior persists even after supervised fine-tuning (SFT) by which the base model is further trained on expert trajectories. Figure 2 : Exploration Failure of Base and SFT Models. ( Left ) Trajectory diversity measured by entropy and the distinct

Read the complete text