Efficient Learning for Product Attributes with Compact Multimodal Models

Mandar Kulkarni · 2025

Read Efficient Learning for Product Attributes with Compact Multimodal Models by Mandar Kulkarni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19679.txt.

Introduction

Sellers need each product photo tagged with the right traits, and Mandar Kulkarni's 2025 paper tackles that job for online shops. Hand labels cost too much at scale, so Kulkarni works with compact vision language models of two to three billion parameters. The method starts from a small labeled set built with an outside annotation service, then trains small adapter modules on it.

The adapters then face unlabeled listings. For each item the model drafts several reasoned answers, and the answers that agree most are treated as preferred while the rest count as dispreferred. A preference training method called DPO learns from that split, and the round repeats with the updated model. Across twelve shop categories, the method beat plain supervised training and kept gaining as more unlabeled data came in, which shows a large pool of raw listings can lift accuracy cheaply.

Catalog record

CREATORMandar Kulkarni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-19679

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Learning for Product Attributes with Compact Multimodal Models Mandar Kulkarni Abstract Image-based product attribute prediction in e-commerce is a crucial task with numerous applications. The supervised fine-tuning of Vision Language Models (VLMs) faces significant scale challenges due to the cost of manual or API based annotation. In this paper, we investigate label-efficient semi-supervised fine-tuning strategies for compact VLMs (2B–3B parameters) that leverage unlabeled product listings through Direct Preference Optimization (DPO). Beginning with a small, API-based, annotated, and labeled set, we first employ PEFT to train low-rank adapter modules. To update the adapter weights with unlabeled data, we generate multiple reasoning-and-answer chains per unlabeled sample and segregate these chains into preferred and dispreferred based on self-consistency. We then fine-tune the model with DPO loss and use the updated model for the next iteration. By using PEFT fine-tuning with DPO, our method achieves efficient convergence with minimal compute overhead. On a dataset spanning twelve e-commerce verticals, DPO-based fine-tuning, which utilizes only unlabeled data, demonstrates a significant improvement over the supervised model. Moreover, experiments demonstrate that accuracy with DPO training improves with more unlabeled data, indicating that a large pool of unlabeled samples can be effectively leveraged to improve performance. 1 Introduction Vertical Attribute Description Allowed values suitcase pattern Pattern refers to a repeated decorative sketch that is printed or engraved on a product. Character,Checkered,Colorblock,Printed, Solid, Striped,Textured,Abstract,Tropical, Geometric-Print,Graphic-Print,Floral-Print,Polka-Print, Animal-Print kurta detail placement Detail Placement refers to the print or design placed on particular place of the garment. All-Over, Back, Waist, Hemline, Neckline, None,Side, Sleeve, Slits, Yoke,Chest,Cuff,Pocket,Front-Panel Table 1 : Examples of vertical specific attribute, description and allowed values Figure 1 : Images from the suitcase vertical. Typically e-commerce images contain various views of the same product at different scales. Image-based attribute prediction is a crucial task in the e-commerce domain, with numerous practical applications, including automated listing assistance and quality control. Unlike open - ended generation tasks, attribute prediction is inherently objective: the values produced by the model can be directly verified against ground truth or image content. Vision–Language Models (VLMs) have recently achieved state-of-the-art performance on cross-modal benchmarks by fusing visual features with textual embeddings. However, large-scale multimodal architectures incur prohibitive computational and memory costs, which restrict their use in latency-sensitive applications common in e-commerce operations. Supervising the fine-tuning of such large VLMs is impractical in the e-commerce setting. Modern catalogs host millions of items across diverse categories—or “verticals”—and manual labeling at this scale is infeasible. Moreover, seller - supplied attribute values (e.g., “color,” “occasion,” “ideal_for”) are often noisy and require extensive manual quality control. Although large VLM APIs can generate labels automatically, the operational expense becomes prohibitive at catalog scale. In contrast, unlabeled images from product listings are abundant and represent a valuable resource if harnessed effectively. In this work, we investigate the viability of small VLMs (2–3 billion parameters) for image - based attribute prediction in e - commerce. We evaluate two Qwen-family models—Qwen2 - VL - 2B - Instruct and Qwen2.5 - VL - 3B - Instruct—by adapting them to the attribute-prediction task. To minimize computational overhead, we employ Parameter-Efficient Fine-Tuning (PEFT) [ 3 ] , updating only the low-rank adapter weights. We first use large VLM APIs to generate reasoning - and - attribute labels for a small subset of product listings. We fine - tune adapters on this labeled seed set, then perform adapter tuning on 5,000 unlabeled listings (each with up to five images) using the DPO objective. Prior research has shown that self-consistency strongly correlates with model uncertainty and accuracy on chain-of-thought tasks [ 7 , 5 ] . Building on this insight, we explore the Direct Preference Optimization (DPO) approach for Unsupervised Fine-Tuning (UFT) using unlabeled data. DPO [ 4 ] replaces conventional reinforcement learning objectives with a convex loss that directly aligns model log probabilities with observed preference data. For each unlabeled example, we generate multiple reasoning-and-answer chains and identify the most consistent prediction to designate "preferred" versus "dispreferred" outputs. We then fine-tune the model using DPO loss on these pseudo-labels. We iterate this process by regenerating pseudo-labels with the updated model, followed by DPO fine-tuning. To avoid updates based on predictions of high uncertainty, we employ a consistency confidence threshold, defined as the fraction of predictions that match the most consistent prediction. The model is updated only for predictions that exceed this threshold. For training with unlabeled data, we also experimented with the Self-Learning (SL) objective, which showed promising results on high-capacity LLMs [ 1 ] . We implement a pseudo-labeling loop in which the VLM generates candidate labels for unlabeled listings, filters them based on the confidence threshold, and retrains itself only on these preferred generations. However, we observed that the SL objective degrades the accuracy in small VLM settings. Our train and test datasets span twelve e-commerce verticals, each defined by a set of visual attributes (e.g., color, pattern, occasion, ideal_for). Experimental results demonstrate that DPO - based fine - tuning on pseudo - labeled data yields significant improvements in attribute - prediction accuracy. The high accuracy lifts are observed even when pretrained checkpoints are directly updated using DPO loss on unlabeled data. We observe that the accuracy with DPO training improves with an increasing amount of unlabeled data, indicating that a large pool of unlabeled samples can be effectively leveraged to enhance performance. 2 Data description Attribute Question prompt Answer prompt type what is the most suitable value for type for this suitcase . Type refers to the classification of the product based on its use and structure.Options: [’Cabin & Check-in Set’, ’Check-in Suitcase’, ’Cabin Suitcase’, ’Overnighter & Briefcase’] <reasoning>The suitcase appears to be a standard check-in suitcase based on its size and design, which is suitable for larger travel needs.</reasoning><answer>Check-in Suitcase</answer> locking_mechanism what is the most suitable value for locking_mechanism for this suitcase . Locking Mechanism refers to the locking mechanism of the product.Options: [’None’, ’Number Lock’, ’TSA Lock’, ’Key Lock’, ’Combination Lock’, ’Smart Lock’] <reasoning>The suitcase features a 3-digit combination lock as indicated in the third image, which clearly shows the locking mechanism.</reasoning><answer>Combination Lock</answer> Table 2 : Sample training examples, prepared using prompt formats, for two attributes of a listing from Suitcase vertical System Prompt : Respond in the following format:<reasoning> … </reasoning><answer> … </answer> User Prompt : what is the most suitable value for <attribute name> for >this <vertical>. <attribute description> Options: <allowed values> Answer prompt : <reasoning> reason </reasoning><answer> answer </answer> Table 3 : Prompt formats used for the VLM training E - commerce catalog data typically comprises multiple product images captured from different viewpoints, organized hierarchically by product category (or “vertical”) down to individual listings. Each vertical defines its own set of attributes that are displayed on the product page—for example, footwear verticals like “sandal” or “slipper flip - flop” may include attributes such as heel pattern or color, whereas the “suitcase” vertical has attributes such as locking mechanism, number of wheels. For our experiments, we selected product listings from twelve distinct verticals—sandal, ethnic set, kurta, shoe, slipper flip - flop, trouser, t - shirt, top, shirt, suitcase, sari, and watch. To help sellers list products efficiently, e-commerce platforms provide vertical-dependent attribute descriptions and sets of allowed values, enabling quick and standardized input.

Read the complete text