From Pixels to Posts: Retrieval-Augmented Fashion Captioning and Hashtag Generation Journal: Expert Systems with Applications
Moazzam Umer Gondal, Hamad Ul Qudous, Daniya Siddiqui, Asma Ahmad Farhan · 2025
Read From Pixels to Posts: Retrieval-Augmented Fashion Captioning and Hashtag Generation Journal: Expert Systems with Applications by Moazzam Umer Gondal, Hamad Ul Qudous, Daniya Siddiqui, Asma Ahmad Farhan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19149.txt.
Introduction
Moazzam Umer Gondal, Hamad Ul Qudous, Daniya Siddiqui, and Asma Ahmad Farhan build a system that writes fashion captions and hashtags from product photos. Plain image captioners often miss details such as fabric or fit and struggle on new styles. Their pipeline grounds the text in detected facts before any writing happens.
A YOLO detector finds each garment, color clustering extracts dominant shades, and a CLIP-FAISS search infers fabric and gender from a product index. Those facts, plus style examples, guide a language model to write the caption. Compared with a fine-tuned BLIP baseline, the retrieval pipeline covers attributes better and generalizes more broadly.
Catalog record
| CREATOR | Moazzam Umer Gondal, Hamad Ul Qudous, Daniya Siddiqui, Asma Ahmad Farhan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-19149
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Pixels to Posts: Retrieval-Augmented Fashion Captioning and Hashtag Generation Journal: Expert Systems with Applications Moazzam Umer Gondal, Hamad Ul Qudous, Daniya Siddiqui, Asma Ahmad Farhan Abstract This paper introduces the retrieval-augmented framework for automatic fashion caption and hashtag generation, combining multi-garment detection, attribute reasoning, and Large Language Model (LLM) prompting. The system aims to produce visually grounded, descriptive, and stylistically interesting text for fashion imagery, overcoming the limitations of end-to-end captioners that have problems with attribute fidelity and domain generalization. The pipeline combines a YOLO-based detector for multi-garment localization, k-means clustering for dominant color extraction, and a CLIP-FAISS retrieval module for fabric and gender attribute inference based on a structured product index. These attributes, together with retrieved style examples, create a factual evidence pack that is used to guide an LLM to generate human-like captions and contextually rich hashtags. A fine-tuned BLIP model is used as a supervised baseline model for comparison. Experimental results show that the YOLO detector is able to obtain a mean Average Precision ([email protected]) of 0.71 for nine categories of garments. The RAG-LLM pipeline generates expressive attribute-aligned captions and achieves mean attribute coverage of 0.80 with full coverage at the 50% threshold in hashtag generation, whereas BLIP gives higher lexical overlap and lower generalization. The retrieval-augmented approach exhibits better factual grounding, less hallucination, and great potential for scalable deployment in various clothing domains. These results demonstrate the use of retrieval-augmented generation as an effective and interpretable paradigm for automated and visually grounded fashion content generation. 1 Introduction Fashion images on social media frequently depict multiple garments styled together and captured in unconstrained conditions, where composition, lighting, and pose are optimized for human appeal rather than machine perception. The central problem is to automatically produce a caption and a set of hashtags that are faithful to the visual evidence: the text must name the detected garment categories, correctly reflect salient attributes such as dominant colors and likely fabric, and adopt language that suits social platforms and brand tone. Despite steady progress in generic image captioning, recent surveys document persistent difficulty with fine-grained attribute grounding and multi-object compositionality, which leads to omissions or hallucinated object–attribute pairings [ 25 , 28 ] . For example, when two similarly textured garments co-occur, captioners may collapse them into a single mention or attach the wrong color term to the wrong item; when accessories are partially occluded, they may be ignored or spuriously introduced [ 13 ] . In creator and brand workflows, such errors translate into captions that read well but fail to reflect what is actually in the photo, and into hashtags that skew generic rather than representative of color, material, or audience [ 7 , 12 ] . The significance of this issue is that captions and hashtags serve as light-weight, consumer-facing metadata and have a direct impact on search, recommendations, and audience targeting in fashion ecosystems. Offering industry-facing analyses, it is claimed that social streams are one of the most important leading indicators of trend dynamics and merchandising signals, and this has a definite implication on planning and content strategy [ 2 ] . Tag suggestion studies show that context-aware and structured tags enhance discoverability and engagement through matching user intent with platform discovery and ranking of tags [ 12 ] . A similar role is played by retrieval throughout the fashion pipelines: product discovery relies on visual similarity search, near-duplicate curation, and attribute propagation, and retrieval offers a natural interface between perception and language supervision when explicit labels are sparse or noisy [ 11 ] . In practice, the brands have to work both in the studio catalog imagery and in-the-wild social posts, a deployable solution needs to have strong perception of a variety of garments and local colors, and controlled generation that can give short and style-consistent copy and balanced mix of broad and niche tags, at latencies that fit within real-time or near-real-time content processes. Literature shows a trajectory from CNN–RNN encoder–decoders toward transformer-based vision–language models. Surveys catalogue advances in attention mechanisms, large-scale pretraining, and decoding strategies, while noting weaknesses in attribute specificity and compositional generalization that surface in complex scenes [ 25 , 28 ] . Methods that enrich the visual representation with relational structure (e.g., scene graphs) mitigate certain error modes by making object–relation patterns explicit; nonetheless, they often struggle to bind the correct attribute tokens to the correct instances when similar items co-occur or partially occlude one another [ 13 ] . In fashion-specific settings, works that explicitly align detected visual attributes with language—conditioning generation on item categories and attributes—report improved faithfulness and reduced drift from the image content, suggesting attribute grounding is central to domain realism [ 23 ] . Complementary research treats captioning as a retrieval-informed task, retrieving semantically close exemplars or summaries to stabilize wording, reduce hallucination, and better reflect the target register and platform constraints [ 19 ] . On the social side, studies of caption and hashtag recommendation reinforce the value of controlled phrasing and structured tags for visibility, intent-signaling, and downstream analytics [ 7 , 12 ] . Together, these threads motivate designs that unify multi-object perception, explicit attribute extraction, and retrieval-informed generation, rather than relying on a single monolithic captioner. Addressing this problem in practice introduces several challenges that shape the design space. First, multi-garment perception is intrinsically difficult: garments overlap, self-occlude, and deform, and recall must be preserved without admitting spurious detections that pollute downstream text. Analyses of modern one-stage detectors highlight sensitivity to confidence thresholds, non-maximum suppression, and class imbalance, underscoring a precision–recall trade-off that must be carefully managed in fashion scenes [ 17 , 15 ] . Second, attribute fidelity depends on localized evidence: dominant colors should be computed on garment crops rather than the whole image to avoid background bias, and fine-grained cues such as likely fabric or audience frequently require auxiliary signals beyond the detector’s category label. Fine-grained garment segmentation and fashion classification studies emphasize the value of instance- or part-level delineation to recover attributes that are visually subtle but semantically important [ 10 , 3 ] . Third, domain shift between studio catalog photos and social posts degrades out-of-domain performance; cross-domain evaluations regularly report substantial drops without explicit adaptation or retrieval-based conditioning [ 22 ] . Finally, deployment requires robustness testing and diagnostics to surface failure modes such as attribute hallucination, missed detections, or brittle decoding behaviors, so that thresholds and safeguards can be tuned for real settings [ 29 ] . We address these challenges with a retrieval-augmented formulation that modularizes perception and generation, passing structured evidence between them. Our approach integrates multi-object detection to identify garment instances, localized color estimation to anchor color words, and retrieval against a curated catalog to supply stable attribute cues and stylistic examples. This modular framework allows the system to generate fluent captions and diverse hashtags by conditioning on a compact evidence pack that enumerates detected categories, localized colors, and retrieved attributes. Importantly, the retrieval step serves as a stability prior, reducing attribute hallucination and ensuring diversity in linguistic output without requiring the language generator to be retrained. To the best of our knowledge, this work is the first to focus on South Asian fashion using a pipeline that combines multi-garment