Agent Journey Beyond RGB: Unveiling Hybrid Semantic-Spatial Environmental Representations for Vision-and-Language Navigation
Xuesong Zhang, Yunbo Xu, Jia Li, Zhenzhen Hu, Richnag Hong · 2024
Read Agent Journey Beyond RGB: Unveiling Hybrid Semantic-Spatial Environmental Representations for Vision-and-Language Navigation by Xuesong Zhang, Yunbo Xu, Jia Li, Zhenzhen Hu, Richnag Hong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06465.txt.
Description
Xuesong Zhang and colleagues present SUSA, an architecture that helps navigation agents understand indoor environments, in a paper published in two thousand twenty-four. Vision-and-language navigation asks an agent to follow spoken or written instructions through unseen rooms. People naturally combine word meanings with a sense of spatial layout when doing this.
Earlier systems often just glued extra data types onto image features. SUSA instead works at two scales. One module generates descriptions of each view to support local action choices. Another builds a depth-based exploration map along the agent's path, capturing global layout. Experiments show this layered enrichment improves navigation performance substantially over prior approaches.
Catalog record
| CREATOR | Xuesong Zhang, Yunbo Xu, Jia Li, Zhenzhen Hu, Richnag Hong |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06465