Do Visual Imaginations Improve Vision-and-Language Navigation Agents?

Akhil Perincherry, Jacob Krantz, Stefan Lee · 2025

Read Do Visual Imaginations Improve Vision-and-Language Navigation Agents? by Akhil Perincherry, Jacob Krantz, Stefan Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16394.txt.

Introduction

Akhil Perincherry and co-authors test whether showing navigation agents imagined views of their goals helps them follow spoken directions, in a 2025 paper. Such agents must find their way in unseen places from language alone. Landmarks named in the directions could serve as visual cues if pictured in advance.

An image generation model turns landmark phrases from split instructions into pictures. Those pictures are fed to the agent as an extra sense, and a training loss ties each picture to its words. Across agents, success rates rose by about a point, with path efficiency gains as well. The authors say imagined landmarks sharpen visual grounding beyond words alone. Code and data are shared.

Catalog record

CREATORAkhil Perincherry, Jacob Krantz, Stefan Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16394

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Visual Imaginations Improve Vision-and-Language Navigation Agents? Akhil Perincherry, Jacob Krantz, Stefan Lee Abstract Vision-and-Language Navigation (VLN) agents are tasked with navigating an unseen environment using natural language instructions. In this work, we study if visual representations of sub-goals implied by the instructions can serve as navigational cues and lead to increased navigation performance. To synthesize these visual representations or “imaginations”, we leverage a text-to-image diffusion model on landmark references contained in segmented instructions. These imaginations are provided to VLN agents as an added modality to act as landmark cues and an auxiliary loss is added to explicitly encourage relating these with their corresponding referring expressions. Our findings reveal an increase in success rate (SR) of ∼ \sim 1 point and up to ∼ \sim 0.5 points in success scaled by inverse path length (SPL) across agents. These results suggest that the proposed approach reinforces visual understanding compared to relying on language instructions alone. Code and data for our work can be found at https://www.akhilperincherry.com/VLN-Imagine-website/ . 1 Introduction In this work, we examine whether providing visual imagery corresponding to described landmarks improves the performance of agents following natural language navigation instructions. Consider the natural language navigation instruction presented in Figure 1 which asks an agent to “ Go straight, take a left at the pool table to enter the kitchen. Walk to the bedroom and stop ”. This instruction provides unconditional action directives like “ go straight ” but also frequently conditions the given directions on visual landmarks in the scene such as the pool table, kitchen, and bedroom. Standard approaches to vision-and-language navigation tasks rely on learned cross-modal alignment mechanisms to implicitly associate these noun phrases with their visual referents during navigation. However, text-to-image generation models have improved to the point that producing imagery matching the semantics of these visual references prior to navigation is plausible. We show three such generations for our running example in Figure 1 . Drawing an analogy to the substantial work in cognitive science on the impact of mental imagery on task performance, we refer to these generated images as visual imaginations and study whether providing them in addition to corresponding language-based instructions can improve the performance of vision-and-language navigation agents. Figure 1 : Illustration of visual imaginations. (Top) A natural language instruction specifying sub-goals pool table, kitchen, and bedroom. (Bottom) Visual imaginations of landmarks pool table, kitchen and bedroom referenced by the sub-goals in the instruction. In our work, we study if these visual imaginations generated using text-to-image models can improve performance in VLN. But why might we believe visual imaginations could be beneficial? In short – text-to-image models have broad knowledge, and learning to perform semantic matching in the image domain may be an easier task than language grounding. Associating noun phrases and their visual referents is a sub-task in vision-and-language navigation (VLN) that most works address through extensive pretraining of model components on web-scale image caption datasets – e.g., by initializing image and language encoders with CLIP Radford et al. 2021 models. While this provides a useful warm-start, it is unclear how well the remaining cross-modal components that drive navigation retain these capabilities after downstream VLN training. Downstream datasets may cover only a small portion of relevant objects and visual grounding remains a weakness in many existing VLN models Zhu et al. 2022 ; Yang et al. 2023 . Directly synthesizing visual imaginations using off-the-shelf text-to-image models offers alternative training and inference mechanisms. Synthesized images have known correspondences to noun phrases in instructions that can be leveraged to reinforce visual grounding during downstream task training through auxiliary losses. Further, matching a visual imagination with environment observations reduces to an image-to-image matching task – offering the potential to use visual imaginations as a convenient pivot between language and vision. This may prove especially effective for out-of-distribution landmarks that have limited or no support during downstream task training. For example, novel instructions referencing “butterfly sculptures” or “Pulp Fiction posters” are unlikely to be well-grounded in the training set but produce semantically relevant visual imaginations that could then be matched to observations. Though we do not imply any direct connections, our work is also inspired by cognitive science studies which indicate mental imagery can serve as valuable cues in addition to linguistic stimuli. Several works supporting dual-coding theory Paivio 1968-2 ; Paivio 1965-2 ; Paivio et al. 1968 find that concrete imagery tends to be more effective for certain types of concept learning compared to linguistic stimuli alone. Further, findings from human and animal studies Reddan et al. 2018 ; Shepard and Metzler 1971 ; Schacter et al. 2007 ; Lai et al. 2023 suggest that imagining snapshots of the environment or potential trajectories before navigating can improve decision making. To investigate our key question, we augment agents with visual imaginations for the popular Room-2-Room (R2R) Anderson et al. 2017 and REVERIE Qi et al. 2020 tasks. We propose a VLN agent agnostic procedure to incorporate visual imaginations as additional inputs and introduce a text-imagination alignment loss to explicitly encourage visual grounding during training. We generate visual imaginations for key noun phrases in instructions that refer to visual landmarks using an off-the-shelf text-to-image generation model Podell et al. 2023 . Applying our technique to two existing models, HAMT Chen et al. 2021 and DUET Chen et al. 2022 , we find modest improvements – R2R validation unseen success increases by roughly 1.0 and 0.6 points respectively. This translates to an improvement of 2 points in success on the test set for our modified DUET model. On REVERIE validation unseen, we improve DUET performance by 1.3 points success and 0.82 points grounding success. Our results suggest that including explicit visual depictions of referred objects can improve performance on vision-and-language navigation tasks. Contributions. We summarize our contributions as: – We develop a pipeline for generating visual imaginations from navigation instructions and synthesize the R2R-Imagine dataset to enable studying the impact of text-to-image models on VLN agents. – We propose an agent-agnostic method to incorporate visual imaginations into existing VLN agents and show improved performance for HAMT Chen et al. 2021 and DUET Chen et al. 2022 across R2R and REVERIE. – We provide ablations of modeling decisions to characterize the design space for adding visual imaginations. 2 Related Work Vision-and-Language Navigation (VLN) . Navigating agents that can operate in novel environments using free-form natural language lends itself as personal assistants, field-robots and search-and-rescue agents. Development of VLN agents has been facilitated by photorealistic simulators such as Matterport3D Chang et al. 2017 and paired language-action datasets such as Room-2-Room (R2R) Anderson et al. 2017 and REVERIE Qi et al. 2020 . R2R provides fine-grained instructions conveying low-level directives such as “Go left, then right at the hall”. REVERIE specifies coarse-grained instructions conveying broad high-level goals such as “Adjust the picture by the lamp in the hall”. Progress in VLN has broadly ranged from architectural improvements and data scaling techniques. Architectural improvements include evolution from recurrent systems Anderson et al. 2017 to transformer architectures Hong et al. 2021b ; Chen et al. 2021 ; Chen et al. 2022 . HAMT Chen et al. 2021 uses a hierarchical transformer architecture to encode spatial and temporal information. DUET Chen et

Read the complete text