Agent Journey Beyond RGB: Unveiling Hybrid Semantic-Spatial Environmental Representations for Vision-and-Language Navigation

Xuesong Zhang, Yunbo Xu, Jia Li, Zhenzhen Hu, Richnag Hong · 2024

Read Agent Journey Beyond RGB: Unveiling Hybrid Semantic-Spatial Environmental Representations for Vision-and-Language Navigation by Xuesong Zhang, Yunbo Xu, Jia Li, Zhenzhen Hu, Richnag Hong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06465.txt.

Description

Xuesong Zhang and colleagues present SUSA, an architecture that helps navigation agents understand indoor environments, in a paper published in two thousand twenty-four. Vision-and-language navigation asks an agent to follow spoken or written instructions through unseen rooms. People naturally combine word meanings with a sense of spatial layout when doing this.

Earlier systems often just glued extra data types onto image features. SUSA instead works at two scales. One module generates descriptions of each view to support local action choices. Another builds a depth-based exploration map along the agent's path, capturing global layout. Experiments show this layered enrichment improves navigation performance substantially over prior approaches.

Catalog record

CREATORXuesong Zhang, Yunbo Xu, Jia Li, Zhenzhen Hu, Richnag Hong
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06465

Browse the full library · About the Tower of Babel