Before Smelling the Video: A Two-Stage Pipeline for Interpretable Video-to-Scent Plans Conference: April 13–17, 2026; Barcelona; CCS: Human-centered computing Human computer interaction (HCI)

Kaicheng Wang, Kevin Zhongyang Shao, Ruiqi Chen, Sep Makhsous, Denise Wilson · 2026

Read Before Smelling the Video: A Two-Stage Pipeline for Interpretable Video-to-Scent Plans Conference: April 13–17, 2026; Barcelona; CCS: Human-centered computing Human computer interaction (HCI) by Kaicheng Wang, Kevin Zhongyang Shao, Ruiqi Chen, Sep Makhsous, Denise Wilson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19203.txt.

Introduction

Kaicheng Wang and four co-authors teach video systems to plan smells that match what is on screen. Smell can deepen immersion in media, yet it is rare because scent is hard to author and to sync with moving video. Past scent tools lean on hand set triggers and fixed links between events and odors, which do not scale to open content.

Their two stage pipeline first reads the video and drafts a scent plan a person can inspect. Keeping the plan readable lets a human check it before any scent is made. The work asks whether semantic planning can carry smell design the way it now carries image and sound design.

Catalog record

CREATORKaicheng Wang, Kevin Zhongyang Shao, Ruiqi Chen, Sep Makhsous, Denise Wilson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19203

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Before Smelling the Video: A Two-Stage Pipeline for Interpretable Video-to-Scent Plans Conference: April 13–17, 2026; Barcelona; CCS: Human-centered computing Human computer interaction (HCI) Kaicheng Wang, Kevin Zhongyang Shao, Ruiqi Chen, Sep Makhsous, Denise Wilson Abstract . Olfactory cues can enhance immersion in interactive media, yet smell remains rare because it is difficult to author and synchronize with dynamic video. Prior olfactory interfaces rely on designer triggers and fixed event-to-odor mappings that do not scale to unconstrained content. This work examines whether semantic planning for smell is intelligible to people before physical scent delivery. We present a video-to-scent planning pipeline that separates visual semantic extraction using a vision-language model from semantic-to-olfactory inference using a large language model. Two survey studies compare system-generated scent plans with over-inclusive and naive baselines. Results show consistent preference for plans that prioritize perceptually salient cues and align scent changes with visible actions, supporting semantic planning as a foundation for future olfactory media systems. Figure 1. We introduce a two-stage video-to-scent planning pipeline that translates visual events in video into structured, human-interpretable scent plans, without generating physical scents. (A) A vision–language model (Gemini 3 Pro) processes uniformly sampled video frames to extract time-aligned visual semantics, producing a structured visual timeline without olfactory assumptions. (B) A large language model (GPT-5.2) transforms this visual timeline into a structured scent plan by prioritizing olfactorily relevant sources and reasoning about relative intensity and temporal evolution under a fixed odor schema. (C) The output is a temporally organized scent plan designed for future olfactory interfaces, separating semantic planning from physical scent delivery. 1. Introduction Immersion in interactive media is shaped by how sensory channels and interaction mechanics jointly support a felt sense of presence. Video is a rich and expressive medium, yet compared to vision and sound, olfaction remains rare in deployed interactive systems, despite its strong association with emotion and memory and its demonstrated potential to increase presence in immersive media ( 4 ; 22 ; 21 ) . A core reason is that scent is difficult to integrate into dynamic media in a principled way. Scent hardware is harder to control than pixels or audio, scents linger and mix, and designers often lack reliable workflows for specifying when and what to release ( 17 ; 13 ; 6 ) . These challenges are especially pronounced for video. Unlike static scenes, videos evolve continuously, with rapid shifts in objects, actions, and context. Because olfaction has distinct temporal dynamics, including onset, offset, and washout, “when” to present a scent can matter as much as “what” to present ( 23 ) . Prior olfaction-enhanced multimedia systems therefore commonly rely on manually scripted scent timelines per clip. While effective for curated demonstrations, this approach does not scale to diverse or user-generated video ( 12 ; 23 ) . As a result, the bottleneck is often not generating scent, but deciding which video elements are olfactorily salient and translating them into a time-varying control plan that respects human tolerance for synchronization mismatch and the practical constraints of scent delivery ( 13 ; 26 ) . In parallel, recent vision–language models (VLMs) and video-language systems make it increasingly feasible to extract structured semantics from video, including objects, activities, and temporally grounded events ( 18 ; 35 ; 25 ) . However, these outputs are typically optimized to describe what happened in a clip, rather than to support downstream sensory decisions. Smell-enhanced video thus requires an additional transformation layer that prioritizes a small subset of olfactorily relevant cues and converts them into a temporally organized scent plan specifying relative timing, intensity, and transitions. In this work, we present an ongoing system exploration that investigates this semantic planning layer as a prerequisite to building end-to-end olfactory media systems in real world. We introduce a two-stage computational video-to-scent support pipeline that (1) extracts purely visual semantics from sampled video frames using VLM and (2) infers structured olfactory representations using a large language model constrained by a fixed odor schema. Rather than generating physical scents, the system produces structured scent plans intended to be intelligible, coherent, and actionable for future olfactory interfaces. To examine whether such semantic-to-scent planning aligns with human expectations, we conduct a controlled user study that evaluates system-generated scent plans without requiring physical scent output. The study compares system-generated plans against over-inclusive and naive baselines, focusing on perceived olfactory relevance, temporal coherence, and anticipated experiential impact when imagined as part of a video viewing experience. The following research questions guide our study: RQ1: How well can a computational system generate scent plans that users perceive as temporally coherent and aligned with dynamic video content? RQ2: Are system-generated scent plans perceived as plausible and non-disruptive when imagined as part of a video viewing experience? 2. Related Work 2.1. Multisensory Augmentation and Immersion in HCI In Human–Computer Interaction, immersion is closely associated with presence and emerges from how sensory channels and interaction mechanics jointly support a felt sense of "being there". Prior work in immersive virtual and augmented reality shows that such systems can heighten users’ sense of presence and embodied ’being-in-place’ ( 7 ; 8 ; 10 ) , and that contextual factors systematically shape these experiences ( 33 ; 16 ) . Beyond vision, auditory design, including background music, spatial audio, and adaptive sound effects, has been shown to influence emotional engagement ( 29 ) , narrative comprehension ( 2 ; 36 ) , and perceived realism ( 15 ; 30 ; 9 ) . Olfaction is strongly linked to emotion and memory, yet remains comparatively underexplored as an interaction modality. Early work such as 4 highlights constraints stemming from hard-to-control hardware and inconsistent odor description schemes, while later perspectives frame smell as a pervasive but still underutilized modality in HCI, motivating olfactory interfaces for affective and cognitive experiences ( 1 ) . Recent CHI research further emphasizes the high friction involved in prototyping and iterating on olfactory experiences, which often require specialized materials, expertise, and ad hoc workflows ( 6 ) . Despite these challenges, studies of immersive media show that even simple scent additions can increase subjective presence and immersion ( 22 ; 21 ) . Compared to vision and sound, which can be tightly coupled with digital media, smell poses distinct challenges for integration ( 6 ; 17 ) . In particular, olfactory cues lack a direct and reliable mapping from dynamic media content to sensory output ( 13 ; 26 ; 27 ) . This limitation is especially pronounced in video-based scenarios, where content evolves continuously with rapid semantic transitions across scenes, objects, and activities ( 24 ; 28 ) . Because olfactory effects exhibit distinct temporal dynamics, the timing of scent presentation can be as consequential as scent selection itself ( 23 ) . Without principled mechanisms for deciding when and what scent should be presented, olfactory augmentation remains difficult to scale beyond manually scripted demonstrations ( 6 ; 28 ) . This gap motivates closer examination of how olfactory interfaces make scent presentation decisions in dynamic, content-driven media. 2.2. Olfactory Interfaces and Smell-Based Interaction Prior HCI research on olfactory interaction has largely treated scent as a discrete, designer-triggered cue. Such systems rely on designer-authored triggers to support functions such as regulating

Read the complete text