ZooClaw-FashionSigLIP2: Distilled Fine-tuning for Robust Fashion Retrieval

Siqiao Xue, Chunxue Xu · 2026

Read ZooClaw-FashionSigLIP2: Distilled Fine-tuning for Robust Fashion Retrieval by Siqiao Xue, Chunxue Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27708.txt.

Introduction

Fashion retrieval needs a vision-language encoder tuned to clothing. Tuning brings a trade-off. Gains on fashion data can cost the base model's broad generalisation. Siqiao Xue and Chunxue Xu present ZooClaw-FashionSigLIP2 in 2026 to resolve that trade-off. The model is a fashion-specialised version of SigLIP2-base. Its recipe has two steps.

First comes full fine-tuning on curated fashion data, with knowledge distillation from the base model. Then WiSE-FT blends the tuned model back with the base model. The recipe beat LoRA tuning, larger billion-parameter backbones, and outside training data. Under fair evaluation, the model led on every benchmark in the suite. The authors also release ZooClaw-Fashion, a new fashion retrieval benchmark. A quality analysis of common benchmarks accompanies it. That analysis exposes structural biases in public ground truth and mitigates them. Weights and evaluation artifacts are open-sourced.

Catalog record

CREATORSiqiao Xue, Chunxue Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27708

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ZooClaw-FashionSigLIP2: Distilled Fine-tuning for Robust Fashion Retrieval Siqiao Xue, Chunxue Xu Abstract Adapting a foundation vision-language encoder to a specialized retrieval task creates a fundamental tradeoff: gains on the target distribution come at the cost of the foundation model’s broad generalization, and fashion retrieval is a stringent instance of this problem. We present ZooClaw-FashionSigLIP2 , a fashion-specialized SigLIP2-base model that resolves this tradeoff with a simple recipe — full fine-tuning with knowledge distillation on curated in-domain data, followed by WiSE-FT ( 22 ) weight interpolation with the base model — and outperforms LoRA, larger backbones (up to 1B parameters), and external training data. Under fair evaluation, ZooClaw-FashionSigLIP2 outperforms all baselines on every benchmark in our suite. In addition, we release ZooClaw-Fashion, a new high-quality fashion retrieval benchmark, and a systematic quality analysis of widely-used benchmarks that exposes and mitigates structural biases in their public ground truth. We open-source the model weights and all evaluation artifacts to facilitate future research. 1 Introduction Fashion image-text retrieval, the task of matching product images to natural language queries and vice versa, is a core component of modern e-commerce search and recommendation systems. In production, the vast majority of search traffic consists of short keyword queries (e.g., “red floral maxi dress”), yet most vision-language encoder (VLE) benchmarks and training recipes focus on detailed natural-language descriptions. A practical fashion retrieval model must excel at both: short queries that users actually type and richer descriptions used in catalog systems. Pre-trained VLEs like CLIP ( 16 ) , SigLIP ( 24 ) , and SigLIP2 ( 18 ) provide strong zero-shot embeddings, but their general-purpose training does not capture the fine-grained visual and textual distinctions that fashion retrieval demands ( 4 ) : subtle differences in neckline, fabric texture, or styling details that determine relevance. Domain-specific fine-tuning is the natural remedy, and models like Marqo-fashionCLIP demonstrate large in-domain gains through contrastive training on fashion data. However, fine-tuning introduces a fundamental tension between in-domain performance and out-of-distribution (OOD) generalization: a production system must serve not only its training catalog but also unseen public catalogs with different query styles, such as structured attribute queries in Fashion200k ( 7 ) or catalog product descriptions in H&M ( 6 ) . In this work, we present ZooClaw-FashionSigLIP2 , a fashion-specialized VLE that resolves this tension through distilled fine-tuning. Through systematic experimentation on the SigLIP2 family, we identify an effective recipe: full fine-tuning with knowledge distillation on curated in-domain data, followed by WiSE-FT ( 22 ) weight interpolation between the fine-tuned and base checkpoints. Neither parameter-efficient methods (LoRA), nor scaling to larger backbones (up to 1B parameters), nor augmenting with external data matches this recipe. Under fair evaluation, ZooClaw-FashionSigLIP2 leads or ties on every metric of every benchmark in our suite, outperforming Marqo-fashionCLIP, Marqo-fashionSigLIP, and the zero-shot SigLIP2 family. For Fashion200k, we adopt a TREC-style pooled re-evaluation with 102,494 held-out judgments, after finding that its public ground truth is biased toward caption-source instance recovery rather than relevance. We open-source the model weights, the ZooClaw-Fashion benchmark, and our pooled evaluation artifacts to facilitate future research. A continuously optimized commercial version is available at https://zoodata.ai/en/api-docs . 2 Related Work VLEs for retrieval. CLIP ( 16 ) established the VLE paradigm for image-text retrieval via contrastive pre-training on web-scale data. Subsequent work improved architecture (SigLIP ( 24 ) , SigLIP2 ( 18 ) ), training efficiency (OpenCLIP ( 9 ) ), and domain specialization (FashionCLIP ( 1 ) ). SigLIP2 replaces the softmax-based InfoNCE loss with a sigmoid loss that eliminates the need for large batch sizes and supports multi-task training. Fashion retrieval. Fashion-specific models include FashionCLIP ( 1 ) , Marqo-fashionCLIP ( 14 ) , and Marqo-fashionSigLIP ( 15 ) . These models improve in-domain performance but often sacrifice generalization. Benchmarks include DeepFashion ( 12 ) , Fashion200k ( 7 ) and LookBench ( 4 ) . Model soups and weight-space ensembling. Model soups ( 21 ) average the weights of multiple fine-tuned models to improve accuracy without increasing inference cost. WiSE-FT ( 22 ) is a special case that interpolates between the zero-shot and a single fine-tuned checkpoint to improve OOD robustness. These methods exploit the observation that fine-tuned models often lie in the same loss basin as the pre-trained model, enabling linear combinations that retain in-domain gains while recovering OOD performance. 3 Method 3.1 Problem Formulation Given a corpus of product images \mathcal{V}=\{v_{1},\ldots,v_{M}\} and a set of text queries \mathcal{T}=\{t_{1},\ldots,t_{N}\} , fashion image-text retrieval aims to learn an embedding space where relevant image-text pairs have high cosine similarity. We consider two retrieval directions: text-to-image (T2I), where a text query retrieves the most relevant images, and image-to-text (I2T), the reverse. A key challenge in practice is that queries vary widely in form—from short keywords (e.g., “red floral maxi dress”) to detailed natural-language descriptions—and the model must generalize across both in-domain and out-of-distribution evaluation scenarios. We adopt a VLE architecture consisting of an image encoder f_{\theta} and a text encoder g_{\phi} that map inputs to a shared d -dimensional embedding space. Our goal is to fine-tune a pre-trained VLE (SigLIP2-base) for fashion retrieval while preserving its OOD generalization. Our approach consists of three stages: ① multi-task contrastive fine-tuning with knowledge distillation ( Section 3.2 ), ② WiSE-FT weight interpolation ( Section 3.3 ), and ③ selection of the interpolation coefficient. 3.2 Multi-Task Contrastive Training Full fine-tuning over LoRA. We use full fine-tuning rather than parameter-efficient methods such as LoRA. Full-rank updates provide the capacity needed to absorb the multi-task objective and the distillation regularizer (introduced below), and our ablations confirm that no LoRA configuration matches full fine-tuning on our suite (see Section 5.2 ). Training objective. We train with a Generalized Contrastive Loss (GCL) ( 25 ) that extends InfoNCE ( 19 ) by incorporating graded relevance scores, allowing the model to leverage soft labels rather than treating all non-matching pairs as equally negative. Given a batch of N image-text pairs \{(v_{i},t_{i},r_{i})\} where r_{i}\in[0,1] is the relevance score of pair i , the text-to-image loss is: \mathcal{L}_{\text{t2i}}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(\text{sim}(t_{i},v_{i})/\tau)}{\sum_{j=1}^{N}w_{ij}\cdot\exp(\text{sim}(t_{i},v_{j})/\tau)}, (1) where \text{sim}(\cdot,\cdot) denotes cosine similarity, \tau is a learnable temperature, and w_{ij}=1-r_{j}\cdot\mathbb{1}[i\neq j] downweights in-batch negatives drawn from pairs that are themselves highly relevant matches — such items are likely false negatives and should not be pushed apart with full strength. The image-to-text counterpart \mathcal{L}_{\text{i2t}} is defined analogously by exchanging the roles of v and t , and the per-task contrastive loss is \mathcal{L}_{k}=\mathcal{L}_{\text{t2i}}+\mathcal{L}_{\text{i2t}} . Multi-task formulation. As noted in Section 3.1 , production retrieval must handle both short keyword queries and longer attribute-rich queries. We address this with two contrastive tasks — short-query and long-query retrieval — combined as \mathcal{L}_{\text{con}}=\lambda_{s}\mathcal{L}_{\text{short}}+\lambda_{l}\mathcal{L}_{\text{long}} with \lambda_{l}{=}1.0 and \lambda_{s}{=}0.5 . Construction of the short and long queries is detailed in Section 4 . Each task already optimizes both retrieval directions through \mathcal{L}_{k} , so we do not define separate image-to-text tasks. Knowledge distillation. Fine-tuning on in-domain data inevitably shifts the learned representations away from the pre-trained model, degrading OOD generalization ( 10 ) . To mitigate this, we apply Learning without Forgetting (LwF) ( 11 ) on the image encoder, which has been shown effective for continual learning in VLEs ( 27 ) . A frozen copy of the base model

Read the complete text