PRISM: Product Retrieval In Shopping Carts using Hybrid Matching
Arda Kabadayi, Jiajing Chen, Senem Velipasalar · 2025
Read PRISM: Product Retrieval In Shopping Carts using Hybrid Matching by Arda Kabadayi, Jiajing Chen, Senem Velipasalar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14985.txt.
Introduction
Arda Kabadayi, Senem Velipasalar, and Jiajing Chen present PRISM for finding products in shopping cart photos in a 2025 computer vision paper. Retail retrieval is harder than generic image search: rival brands look nearly identical, and shopper photos come from odd angles compared with catalog shots. Foundation vision-language models often miss the small local differences that separate products.
PRISM combines the strengths of two approaches in three stages. A vision-language model first pulls a shortlist of the most similar catalog items, narrowing the search. Pixel-wise matching, accurate but slow, then runs only on that shortlist to settle fine distinctions. The hybrid keeps the speed of learned retrieval and the precision of direct image comparison, a practical balance for checkout and inventory applications.
Catalog record
| CREATOR | Arda Kabadayi, Jiajing Chen, Senem Velipasalar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-14985
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PRISM: Product Retrieval In Shopping Carts using Hybrid Matching Arda Kabadayi, Jiajing Chen, Senem Velipasalar Abstract Compared to traditional image retrieval tasks, product retrieval in retail settings is even more challenging. Products of the same type from different brands may have highly similar visual appearances, and the query image may be taken from an angle that differs significantly from view angles of the stored catalog images. Foundational models, such as CLIP and SigLIP, often struggle to distinguish these subtle but important local differences. Pixel-wise matching methods, on the other hand, are computationally expensive and incur prohibitively high matching times. In this paper, we propose a new, hybrid method, called PRISM, for product retrieval in retail settings by leveraging the advantages of both vision-language model-based and pixel-wise matching approaches. To provide both efficiency/speed and fine-grained retrieval accuracy, PRISM consists of three stages: 1) A vision-language model (SigLIP) is employed first to retrieve the top 35 most semantically similar products from a fixed gallery, thereby narrowing the search space significantly; 2) a segmentation model (YOLO-E) is applied to eliminate background clutter; 3) fine-grained pixel-level matching is performed using LightGlue across the filtered candidates. This framework enables more accurate discrimination between products with high inter-class similarity by focusing on subtle visual cues often missed by global models. Experiments performed on the ABV dataset show that our proposed PRISM outperforms the state-of-the-art image retrieval methods by 4.21% in top-1 accuracy while still remaining within the bounds of real-time processing for practical retail deployments. 1 Introduction Figure 1 : Comparison of retrieval approaches: (a) Matching of feature embeddings from a Vision-Language Model (VLM); (b) keypoint matching for finer-grain correspondence, (c) high-level overview of our hybrid PRISM. With significant advancements in AI foundation models, such as CLIP [ 8 ] and SigLIP [ 15 ] , image matching and retrieval have achieved substantial progress recently. These vision-language models have demonstrated strong performance in extracting global semantic features from images, which allows them to find the top-k most semantically similar images in large galleries. While these models are powerful and flexible, they struggle in specialized fields, such as medical imaging [ 18 ] , where the data differs significantly from natural images. Product retrieval in retail world is another application area of image matching/retrieval, but it presents the following additional challenges: (1) High inter-class similarity: different products (e.g. from different brands) can have highly similar visual appearance. For example yogurt containers, cereal boxes, beverage bottles or soda cans often share similar shapes, colors, and packaging style; (2) Varying image viewpoints: the viewing angle of the query/customer image may differ significantly from that of the catalog/gallery images in the system; (3) Domain gap: captured query images from a certain sensor may differ from the catalog images, e.g. in terms of resolution and lighting; (4) Occlusions: customer/query images can be severely occluded blocking identifiable portions of the product. These factors introduce substantial difficulty in reliably retrieving the exact product from large catalogs, especially when subtle differences, such as shapes of the contents of a box (e.g. penne vs. fusilli pasta) are critical for identification. Since VLMs focus on global features, they often struggle to distinguish these subtle but important local differences. Thus, global vision-language models alone cannot fully overcome the aforementioned challenges. Keypoint matching algorithms help address these challenges by enabling fine-grained, pixel-level comparisons that focus on detailed local features. However, their computational cost makes them impractical for directly comparing a query image against every image in very large product galleries. In addition, background feature points can distract and degrade the performance of keypoint matching without additional guidance. Fig. 1 (a) and Fig. 1 (b) illustrate the difference between the VLM-based retrieval using image feature embeddings and fine-grained keypoint matching. VLM-based retrieval uses cosine similarity as the comparison metric, whereas keypoint matching employs a matching algorithm on the entire set of detected keypoints. Motivated by these and considering the strengths and weaknesses of the VLM- and keypoint-based methods, we propose a new, hybrid retrieval framework, referred to as the PRISM, for shopping cart-mounted camera systems in retail environments. Fig. 1 (c) shows a high-level overview of PRISM, which takes advantage of both feature embedding matching and pixel-wise matching. The design of PRISM is based on the extensive experiments we performed on various VLM- and keypoint-based models. As will be discussed below, SigLIP is particularly good at choosing a subset of gallery images that are similar to the query, but struggle in correctly ranking them. Keypoint matching algorithms, on the other hand, are better at ranking. Equipped with these observations, PRISM is designed to have three-stages. In the first stage, SigLIP is used to extract semantic embeddings from the query image and retrieve the top-35 semantically most similar candidates from a product gallery. This step takes advantage of the strengths of SigLIP, efficiently narrows down the search space, and preserves high recall while significantly reducing the computational overhead. In the second stage, to mitigate the background noise potentially distracting keypoint matching, YOLO-E [ 14 ] is employed to segment product regions in both query and candidate gallery images ensuring that comparisons focus only on the relevant object. Finally, the third stage utilizes LightGlue [ 5 ] , a robust local feature matcher, to perform detailed keypoint-level matching between the segmented product regions. This combination enables more accurate discrimination between products with high inter-class similarity by focusing on subtle visual cues often missed by global models. Thus, our proposed approach provides substantial improvement in top-1 accuracy of the fine-grained product retrieval in real-world retail scenarios. PRISM is designed with real-time performance in mind, making it suitable for deployment in fast-paced retail environments. Instead of relying only on global feature extractors or exhaustive pairwise matching, our method combines semantic filtering, product segmentation, and local feature matching for efficient and accurate retrieval. The contributions of this work include the following: • We propose a new, three-step product retrieval system, PRISM, which combines the strengths of VLM- and keypoint matching-based approaches. PRISM first uses a VLM to select a candidate list of similar gallery items, then separates the product region via segmentation, and finally applies detailed local feature matching to compare query and a subset of gallery images more precisely. • We present a system that balances retrieval accuracy with speed, making it practical for in-store applications. • We demonstrate the importance of segmentation masks in reducing background noise, thereby improving pixel-level keypoint matching accuracy. • We provide comprehensive experiments showing the role and contribution of each of the proposed three stages. • We validate our approach on a real-world dataset, collected from shopping cart-mounted cameras, and show that it outperforms state-of-the-art VLM-based retrieval models, achieving significant gains in Top-1 accuracy. Our experiments confirm that combining semantic-level retrieval with pixel-level local matching can bridge the gap between efficient search and accurate discrimination in fine-grained product recognition. 2 Related Work 2.1 Image Retrieval Early image retrieval methods relied on handcrafted local features. SIFT [ 6 ] and SURF [ 7 ] capture distinctive keypoints for fine-grained matching. These approaches were effective but struggled with scalability and robustness to variations. With the rise of deep learning, Convolutional Neural Networks (CNNs) became popular for data-driven feature extraction,