See & Sniff: Learning Visuo-Olfactory Representations
Seongyu Kim ∗, Seungwoo Lee ∗, Hyeonggon Ryu, Joon Son Chung, Arda Senocak · 2026
Read See & Sniff: Learning Visuo-Olfactory Representations by Seongyu Kim ∗, Seungwoo Lee ∗, Hyeonggon Ryu, Joon Son Chung, Arda Senocak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27307.txt.
Introduction
Kim and colleagues teach machines to link what things look like with how they smell. Multimodal models pair vision with language, sound, and touch, but smell has lacked paired data. Smell-only samples can therefore be paired with matching web images by category, building the SmellNet-V dataset without costly joint collection. In 2026, the See and Sniff framework learns joint sight-smell representations from that data.
Dense local alignment ties image regions to odour signals during self-supervised training. Smell saliency maps fall out naturally, pointing at odour sources inside a scene. A pixel-level smell localisation task and benchmark are introduced for scoring. Smell classification from smell alone beats smell-only baselines by a clear margin in the reported results. Cross-modal retrieval and localisation also generalise. Machine perception gains a missing sense with a workable data recipe.
Catalog record
| CREATOR | Seongyu Kim ∗, Seungwoo Lee ∗, Hyeonggon Ryu, Joon Son Chung, Arda Senocak |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27307
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
See & Sniff: Learning Visuo-Olfactory Representations Seongyu Kim ∗, Seungwoo Lee ∗, Hyeonggon Ryu, Joon Son Chung, Arda Senocak Abstract While modern multimodal models integrate vision with language, audio, or touch, olfaction remains largely unexplored due to the lack of paired visuo-olfactory data. We introduce SmellNet-V , a scalable visuo-olfactory dataset built on the insight that odor identity is largely invariant to visual transformations within a semantic category. This allows us to synthetically pair smell-only samples with semantically aligned in-the-wild web images, converting a unimodal olfactory dataset into a cross-modal benchmark without costly co-collection. Building on this dataset, we propose See & Sniff , a self-supervised framework that learns joint visuo–olfactory representations via dense local alignment and naturally produces smell saliency maps for spatial grounding of odor sources. We further introduce pixel-level smell localization task and a benchmark for evaluation. Our method surpasses smell-only baselines by 7% in smell classification from smell alone and generalizes to cross-modal retrieval and smell localization, establishing visuo-olfactory learning as a new direction in multimodal perception. Figure 1 : What can See & Sniff do? We show that See & Sniff , the framework that learns joint visuo-olfactory representations, can handle both unimodal and cross-modal tasks. (Left) Smell localization identifies the locations of smell sources within a visual scene based on input olfactory signals. (Right Top) Cross-modal retrieval demonstrates semantic alignment through bidirectional retrieval. (Right Bottom) Smell classification predicts ingredients from olfactory inputs. † † footnotetext: ⋆ Equal contribution. 1 Introduction Everything has its own odor, which is its soul—tree, flower, soil, rain, burning wood. Helen Keller Modern multimodal AI systems integrate vision with language [ 36 , 21 , 28 , 27 , 53 ] , audio [ 4 , 3 , 40 , 34 , 18 , 15 , 16 ] , and even touch [ 51 , 52 , 50 , 14 , 30 , 24 ] , yet overlook another fundamental sensory modality: smell . Despite its central role in biological perception, olfaction remains minimally explored in AI research, “ a neglected treasure ” [ 23 ] . This gap comes primarily from practical constraints: unlike cameras or microphones, portable olfactory sensors have historically been scarce and confined to laboratory settings, and large-scale public datasets for data-driven learning have been lacking. Recent advances in portable chemical sensing now enable more accessible machine-readable olfactory signals, and the first large-scale smell dataset has very recently been released [ 12 ] . However, it contains only olfactory measurements without paired visual data. Why does visuo-olfactory integration matter? Human vision and olfaction are tightly intertwined sensory systems. When a person sees roasted coffee beans, they anticipate the characteristic bitter and roasted aroma even without smelling it. Conversely, the scent of coffee alone can evoke mental imagery of dark roasted beans or a steaming cup. These correspondences reveal a tight coupling between visual appearance and olfactory identity. Modeling this visuo-olfactory alignment computationally is central to our work. Our objective in this work is to enable a self-supervised model to learn joint representations between visual and olfactory modalities, which fundamentally requires paired visuo-olfactory data. However, such datasets do not currently exist. We address this gap through a key insight : olfactory identity is largely invariant to visual transformations, including changes in lighting, scale, or minor color variations. For example, intra-class variations such as a small red apple and a large red apple emit nearly identical volatile organic compounds (VOCs) and are perceptually categorized by humans under the same odor category, ‘ apple ’. From a representation learning perspective, this suggests that visually diverse instances within a category should share a common semantic odor embedding. Leveraging this invariance, we construct a visuo-olfactory training dataset, SmellNet-V , by synthetically pairing each odor instance from an existing smell-only dataset [ 12 ] with semantically corresponding, in-the-wild open-world web images. This principled pairing strategy transforms a unimodal olfactory dataset into a scalable visuo-olfactory benchmark, enabling large-scale cross-modal alignment without the need for costly real-world paired data collection. Multisensory perception extends beyond simple correspondence. Humans and other animals not only associate odors with visual identities, but also use olfactory cues to navigate and localize odor sources within complex environments [ 7 , 37 ] , i.e . , smell localization . To model this capability, we propose a framework that computes dense similarity maps between local visual and olfactory features, enabling fine-grained alignment. The resulting smell saliency maps highlight image regions expected to emit a given odor, while simultaneously learning holistic cross-modal correspondences. To evaluate smell localization, we introduce the first dataset with pixel-level segmentation aligned with the odor samples. Overall, our proposed training data, SmellNet-V , and self-supervised framework, See & Sniff , establish a new multimodal model for visuo-olfactory perception. The learned joint representations generalize across downstream tasks, including smell classification, cross-modal retrieval, and smell localization. In particular, for smell classification, our model surpasses the smell-only baseline by a significant margin, demonstrating that visuo-olfactory training enhances olfactory representation learning. Together, these contributions introduce new tasks and benchmarks for this emerging modality. Our contributions are summarized as follows: • We construct a visuo-olfactory dataset, SmellNet-V , by leveraging odor invariance within semantic categories to synthetically pair existing smell-only data with in-the-wild visual images. • We propose See & Sniff , a self-supervised multimodal framework that learns joint visuo–olfactory representations via dense local alignment. • We demonstrate that visuo-olfactory training enhances olfactory representation learning, surpassing the smell-only baseline by 7% in smell classification. • We extend visuo-olfactory learning beyond classification and retrieval to spatial grounding by generating smell saliency maps and introducing the first pixel-level smell localization dataset for evaluation. 2 Related Work Machine learning approaches to olfaction have primarily focused on predicting odor perception from chemical and molecular structure. Neural network–based models such as DeepNose [ 46 ] demonstrated that artificial networks can learn structured embeddings of odorant space. Subsequent works extended this direction using graph neural networks [ 39 , 26 , 2 ] , multitask learning frameworks that capture shared representations across related odor categories [ 20 ] , and attention-based aggregation mechanisms built upon molecular representations derived from chemical foundation models [ 22 ] . Other studies have addressed specific challenges such as odor intensity prediction [ 13 ] . Parallel efforts have explored artificial olfactory systems using sensor measurements. Electronic nose based frameworks, such as k-nearest neighbor–based scent classification method [ 32 ] and the eigengraph-based system [ 45 ] , focus on signal-level representation learning and odor classification. While effective, these approaches remain unimodal and are primarily developed in controlled laboratory environments, limiting their scalability to large-scale real-world settings. More recently, SmellNet [ 12 ] introduced the first large-scale real-world smell dataset collected with portable chemical sensors, along with ScentFormer, a transformer-based model for olfactory representation learning. While this represents a significant step toward scalable data-driven machine olfaction beyond laboratory environments, the framework remains unimodal and lacks paired visual information. Building upon SmellNet, we extend its smell-only data to the visual domain by synthetically pairing samples with semantically aligned in-the-wild images, leveraging odor invariance within semantic categories. Whereas ScentFormer focuses on unimodal odor modeling, our framework adopts a two-stream transformer with a dedicated local alignment module