Evaluating Scene-based In-Situ Item Labeling for Immersive Conversational Recommendation
Jiazhou Liang, Yifan Simon Liu, David Guo, Minqi Sun, Yilun Jiang, Scott Sanner · 2026
Read Evaluating Scene-based In-Situ Item Labeling for Immersive Conversational Recommendation by Jiazhou Liang, Yifan Simon Liu, David Guo, Minqi Sun, Yilun Jiang, Scott Sanner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09698.txt.
Introduction
Liang and five co-authors define Immersive Conversational Recommendation, where picked items appear marked with in-situ labels. While picking items is well studied, choosing which details belong in scene labels remains open. The authors sort needs into stated-intent match and unasked needs, and define metrics for picking labels. They benchmark retrieval, language-model, and vision-language-model across fashion, movie, and retail tests. Three limits emerge: methods ignore setting-tied modes, show repeats clear from sight, and poorly foresee unasked needs from chat alone.
Catalog record
| CREATOR | Jiazhou Liang, Yifan Simon Liu, David Guo, Minqi Sun, Yilun Jiang, Scott Sanner |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80514 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.09698
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Scene-based In-Situ Item Labeling for Immersive Conversational Recommendation Jiazhou Liang, Yifan Simon Liu, David Guo, Minqi Sun, Yilun Jiang, Scott Sanner Abstract The growing ubiquity of Extended Reality (XR) is driving Conversational Recommendation Systems (CRS) toward visually immersive experiences. We formalize this paradigm as Immersive CRS (ICRS), where recommended items are highlighted directly in the user’s scene-based visual environment and augmented with in-situ labels. While item recommendation has been widely studied, the problem of how to select and evaluate which information to present as immersive labels remains an open problem. To this end, we introduce a principled categorization of information needs into explicit intent satisfaction and proactive information needs and use these to define novel evaluation metrics for item label selection. We benchmark IR-, LLM-, and VLM-based methods across three datasets and ICRS scenarios: fashion, movie recommendation, and retail shopping. Our evaluation reveals three important limitations of existing methods: (1) they fail to leverage scenario-specific information modalities (e.g., visual cues for fashion, metadata for retail), (2) they present redundant information that is visually inferable, and (3) they poorly anticipate users’ proactive information needs from explicit dialogue alone. In summary, this work provides both a novel evaluation paradigm for in-situ item labeling in ICRS and highlights key challenges for future work. 1 Introduction The growing ubiquity of Extended Reality (XR) has driven rapid adoption across a wide range of domains ( 24 ) . Such technology enables immersive systems with input of egocentric scenes (e.g., head-mounted cameras) (cf. left of Figure 1 ) and information delivery via situated visualization ( 14 ) (e.g., in-situ labels ) within the user’s Point of View (POV). However, immersive interfaces remain largely unexplored in the existing Conversational Recommendation Systems (CRS) literature. Motivated by this gap, we introduce Immersive CRS (ICRS), a new problem that transforms CRS from language-focused interactions to egocentric, in-situ experiences. CRS recommends items from a global catalog in response to seeker requests expressed through natural-language (NL) utterances, modeling seeker needs and preferences as inferred from the conversational context ( 16 ) . In contrast, ICRS recommends items identified from the scene and highlights them directly within the seeker’s POV (cf. right of Figure 1 ). Each recommended item is accompanied by immersive labels , a virtual element that is presented in-situ to referent items, conveying information that is not visually accessible from the physical items ( latent ) to resolve the seeker’s uncertainty and information needs. Figure 1: Left : Seeker’s egocentric view of the scene captured by the immersive system. Right : Recommended items are highlighted and augmented with in-situ immersive labels within the scene. Figure 2: Components in ICRS . Given a conversation prefix and seeker’s egocentric scene (left), candidate items in ICRS are identified via segmentation and enriched with external attributes through visual lookup. ICRS then ranks candidate items and selects in-situ labels (center) that address the seeker’s information needs (e.g., dependability). Recommended items and immersive labels are highlighted and augmented in the scene (right). We illustrate the end-to-end pipeline of ICRS in Figure 2 . While upstream components such as item segmentation and recommendation have been extensively studied ( 10 ; 12 ) , determining and evaluating information in immersive labels, an essential component of ICRS for fully leveraging the capabilities of immersive interfaces, remains largely unexplored. We argue that immersive labels should convey information that (1) justifies why an item aligns with the seeker’s explicit intent as expressed in the current conversation ( Explicit Intent Satisfaction ), and (2) proactively provides decision-relevant information before it is explicitly requested ( Information Needs ). Accordingly, we propose evaluation criteria and metrics for each consideration. We adapt existing CRS datasets with novel, manually curated ground-truth annotations that are specifically designed for immersive label selection and conduct extensive experiments across contemporary methods with IR, LLM, and VLM capabilities in three scenarios: Fashion, Movie, and Retail. Our evaluation results reveal three key limitations of existing methods: 1. They fail to leverage visual and textual information germane to the scenario (e.g., visual cues are more informative in Fashion, while textual metadata are more relevant in Retail). 2. They often present information that is already visually inferable (e.g., color), which offers marginal value w.r.t. users’ information needs. 3. They struggle to preemptively anticipate information that users will need, but have not yet explicitly requested for the recommended item. Together, our findings position label selection for ICRS that fully leverage immersive interfaces as a challenging yet underexplored research problem. 2 Related Work Table 1: Comparison between CRS, Multimodal CRS, and the proposed ICRS across four key dimensions. CRS Multimodal CRS Immersive CRS Input Multi-turn conversation between seeker and recommender Conversation + Egocentric Scene from Seeker’s POV Candidate Items A set of predefined items in the large global catalog Items are identified in the egocentric scene . Item Attributes Textual attributes (e.g., metadata, review) Textual + Predefined multimodal attributes Visual segments from egocentric scene + external attributes identified via item mapping from global corpus Output Recommended items encapsulated within multi-turn utterances Recommended items are highlighted in the egocentric scene with immersive labels to resolve seeker information needs. CRS and Multimodal CRS CRS infer user preferences from multi-turn dialogue and recommend items predefined in a global catalog ( 16 ; 8 ; 10 ; 1 ; 36 ; 35 ; 19 ) , with multimodal variants incorporating item images and metadata to enhance reasoning ( 3 ; 34 ; 7 ) . As shown in Table 1 , ICRS defines a distinct paradigm for immersive interfaces with egocentric, in-situ interaction that is not addressed by prior work. Context-Aware Conversation. Prior work on context-aware conversation largely addresses referential grounding and location-aware utterance generation, as seen in SIMMC and spatial AR dialogue systems ( 13 ; 37 ; 21 ; 15 ) . While effective for linking dialogue to visual or spatial context, these approaches are not designed for preference-oriented recommendation and do not define what item information should be shown via item labels to support recommendation interactions. Situated Visualization In situated visualization ( 14 ) , prior research has addressed the placement and management of visualized information in-situ within the seeker’s POV ( 4 ) . However, these works primarily focus on layout and spatial alignment, rather than the retrieval criteria of information in the label. Image Identification and Attribute Lookup Prior work leverages foundation models like Segment Anything (SAM) ( 12 ) with vision–language pretraining (e.g., CLIP ( 29 ) to identify textually related segments or reverse lookup external attributes from product segmentation in environments (shelf recognition) ( 23 ; 27 ) . These methods provide the raw inputs (cf. Figure 2 ) used by the ICRS labeling approaches we evaluate. 3 Immersive CRS Preliminary Definition The Immersive CRS (ICRS) adapts the CRS definition ( 10 ) . Let \mathcal{U} represent two sides of the conversation: a seeker and a recommender (e.g., human experts or conversational agents ( 18 ) ). Given global candidate item set \mathcal{I}^{\text{glb}} , a conversation can be denoted as a sequence \mathcal{C}=\{(u_{t},s_{t},\mathcal{I}^{\text{glb}}_{t})\}_{t=1}^{T} . At each turn t , either seeker or recommender generates an utterance s_{t}=(\text{w}_{i})_{i=1}^{m} , with each word \text{w}_{i} belonging to the vocabulary, and optionally reference items \mathcal{I}_{t}^{\text{glb}}\subseteq\mathcal{I}^{\text{glb}} .