LaViC : Adapting Large Vision-Language Models to Visually-Aware Conversational Recommendation

Hyunsik Jeon, Satoshi Koide, Yu Wang, Zhankui He, Julian McAuley · 2025

Read LaViC : Adapting Large Vision-Language Models to Visually-Aware Conversational Recommendation by Hyunsik Jeon, Satoshi Koide, Yu Wang, Zhankui He, Julian McAuley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23312.txt.

Introduction

Hyunsik Jeon and four co-authors present a chat-based shopping assistant that uses product images, in a 2025 paper. Text alone misses color, style, and design details that drive fashion and home choices. Their system squeezes each product image into a few compact tokens. A language model then chats using both the dialogue and those tokens.

The squeezing is taught by self-distillation from the model's own image reading. A second stage tunes prompts for the recommendation task. To test the idea, the team built a data set from Reddit chats matched to Amazon listings. Categories leaned visual, such as fashion, beauty, and home goods. The system beat text-only rivals and open vision-language baselines. It matched or topped strong commercial models in their tests.

Catalog record

CREATORHyunsik Jeon, Satoshi Koide, Yu Wang, Zhankui He, Julian McAuley
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23312

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LaViC : Adapting Large Vision-Language Models to Visually-Aware Conversational Recommendation Hyunsik Jeon, Satoshi Koide, Yu Wang, Zhankui He, Julian McAuley Abstract . Conversational recommender systems engage users in dialogues to refine their needs and provide more personalized suggestions. Although textual information suffices for many domains, visually driven categories such as fashion or home decor potentially require detailed visual information related to color, style, or design. To address this challenge, we propose LaViC ( La rge Vi sion-Language C onversational Recommendation Framework), a novel approach that integrates compact image representations into dialogue-based recommendation systems. LaViC leverages a large vision-language model in a two-stage process: (1) visual knowledge self-distillation, which condenses product images from hundreds of tokens into a small set of visual tokens in a self-distillation manner, significantly reducing computational overhead, and (2) recommendation prompt tuning, which enables the model to incorporate both dialogue context and distilled visual tokens, providing a unified mechanism for capturing textual and visual features. To support rigorous evaluation of visually-aware conversational recommendation, we construct a new dataset by aligning Reddit conversations with Amazon product listings across multiple visually oriented categories (e.g., fashion, beauty, and home). This dataset covers realistic user queries and product appearances in domains where visual details are crucial. Extensive experiments demonstrate that LaViC significantly outperforms text-only conversational recommendation methods and open-source vision-language baselines. Moreover, LaViC achieves competitive or superior accuracy compared to prominent proprietary baselines (e.g., GPT-3.5-turbo, GPT-4o-mini, and GPT-4o), demonstrating the necessity of explicitly using visual data for capturing product attributes and showing the effectiveness of our vision-language integration. Our code and dataset are available at https://github.com/jeon185/LaViC . 1. Introduction Conversational recommendation has emerged as a promising framework for e-commerce and digital entertainment, offering an interactive channel for users to express their preferences through natural language ( Christakopoulou et al. 2016 ; Li et al. 2018 ; He et al. 2023 ; Xie et al. 2024 ; He et al. 2025 ) . Unlike traditional recommendation approaches that rely primarily on past interactions, conversational recommender systems aim to find relevant and personalized items by engaging in a dialogue with users. Recently, large language models (LLMs) ( Schulman et al. 2022 ; Touvron et al. 2023a ; Touvron et al. 2023b ; Chiang et al. 2023 ; Brown et al. 2020 ; He et al. 2023 ; Feng et al. 2023 ; Kang et al. 2023 ) have demonstrated a strong ability to interpret user intentions within natural language conversational contexts. Their broad pre-training on extensive tasks allows them to handle complex linguistic features and incorporate specialized domain information when generating responses. This capability has led LLM-based approaches to outperform previous non-LLM methods in conversational recommendation ( He et al. 2023 ; Wang et al. 2023a ; Feng et al. 2023 ) , positioning LLMs as an essential component in the design of current conversational recommender systems. Although text-based recommendation often suffices for many domains, certain product categories (e.g., fashion and home decor) rely heavily on visual features such as color, style, and overall design. Previous visually-aware recommender systems ( McAuley et al. 2015 ; He and McAuley 2016b ; Kang et al. 2017 ) have shown that images can substantially improve performance in visually oriented domains, indicating that purely textual descriptions may miss subtle aesthetic or functional details. For example, a user who requests a “hoodie-like military-style jacket with chest pockets” can narrow the options using textual information, yet verifying the exact silhouette or pocket arrangement depends on visual inspection (Figure 1 ). Thus, incorporating visual information helps resolve ambiguities and improves recommendation quality, making it more likely that a system identifies items aligned with the intended look or function of the user. Figure 1. The Reddit-Amazon dataset contains conversations between a seeker and a recommender. The seeker requests an item recommendation via text, focusing primarily on visual preferences. The item recommended to the seeker by the recommender is linked to detailed item information such as its title and images. To fully integrate visual information into the conversation, one could resort to large vision-language models (VLMs) ( Liu et al. 2023a ; Liu et al. 2024a ; Liu et al. 2024b ) that unify visual and textual inputs. However, such models generally tokenize each image into hundreds or thousands of tokens, incurring a steep computational load if multiple items are analyzed simultaneously. Moreover, naive end-to-end fine-tuning of massive multimodal architectures can lead to overfitting, particularly in specialized domains with limited labeled data. Finally, directly passing all retrieved items as raw image tokens can be prohibitively expensive in real-world deployments, indicating the need for a compact representation of visual content. In this work, we address these challenges with a two-stage framework for visually-aware conversational recommendation built upon VLMs. First, we perform knowledge distillation to compress the high-dimensional encoding of each product image into a minimal set of visual tokens, preserving essential details of appearance while reducing computational overhead. Second, we adopt a recommendation prompt tuning procedure on a distilled vision-language model to jointly process textual queries and these compact visual tokens within a unified generative paradigm. This design balances computational feasibility with the need for detailed visual understanding. Furthermore, to enable rigorous empirical evaluation, we construct a new dataset, called Reddit-Amazon dataset, by aligning Reddit dialogues with Amazon product images, reflecting genuine user queries and realistic visual attributes seldom captured in existing benchmarks. The main contributions of our work are as follows. • We mitigate the token-explosion problem by condensing each product image into a minimal set of visual tokens, preventing the model from processing thousands of image tokens, and thus improving training stability, reducing overfitting, and ultimately enhancing recommendation accuracy. • We propose a prompt-tuning procedure on a distilled vision-language model, enabling it to jointly process textual dialogues and these compressed visual tokens within a unified generative framework for recommendation. • We release Reddit-Amazon , a new dataset consisting of over 19K Reddit conversations (51K turns in total) aligned with Amazon product items. It spans three visually oriented categories (beauty, fashion, and home), linking each recommended item to its title and images. This dataset offers a richer testbed for visually-aware conversational recommendation. 2. Preliminaries 2.1. Problem Statement In the visually-aware conversational recommendation task, we aim to recommend relevant items to a user (the seeker) through a multi-turn dialogue while considering both textual and visual features of the items. Let \mathcal{I} be a set of items, where each item i\in\mathcal{I} has a textual title \mathit{title}_{i} and a single image \mathit{image}_{i} . A vocabulary \mathcal{V} defines the tokens used for text. The conversation is \mathcal{C}=\{s_{t}\}_{t=1}^{T} , where each utterance s_{t} is produced by the seeker or the recommender, drawn from \mathcal{V} . The seeker initially requests recommendations, and at each recommender turn k , the system produces a ranked list \hat{\mathcal{I}}_{k}\subseteq\mathcal{I} that aligns with the true set \mathcal{I}_{k} . This set \mathcal{I}_{k} embodies the user’s current preferences, which may be represented through the conversation. In our setting, we assume |\hat{\mathcal{I}}_{k}|=|\mathcal{I}_{k}|=1 , which means there exists a single ground-truth item and we also aim to recommend a single item. Generative models such as large language models (LLMs) or large vision–language models (VLMs) often produce unconstrained outputs, potentially recommending items that do not exist in \mathcal{I}

Read the complete text