Vega-Video: Integrating Video into the Grammar of Graphics

\authororcid Dominik Winecki0009-0009-8632-3102 and \authororcid Arnab Nandi0000-0002-4138-603X · 2026

Read Vega-Video: Integrating Video into the Grammar of Graphics by \authororcid Dominik Winecki0009-0009-8632-3102 and \authororcid Arnab Nandi0000-0002-4138-603X (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24958.txt.

Introduction

Winecki and Nandi integrate video into the Vega grammar for data graphics. Video is now used with standard data for interactive exploration. They name three kinds of video display: sync, annotation and transformation. A split-signal design links Vega's instant dataflow to video player state. Compile-time detection of scrubbing lets encoding-aware tweaks lift responsiveness up to 4 times. Repurposed VOD protocols transform videos in real time. Updates stay under 200 milliseconds even for multi-hour compilations.

Catalog record

CREATOR\authororcid Dominik Winecki0009-0009-8632-3102 and \authororcid Arnab Nandi0000-0002-4138-603X
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50331
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24958

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vega-Video: Integrating Video into the Grammar of Graphics \authororcid Dominik Winecki0009-0009-8632-3102 and \authororcid Arnab Nandi0000-0002-4138-603X Abstract Video data is increasingly used alongside conventional data for interactive data exploration, necessitating interfaces for exploring and presenting mixed-modality data. However, integrating video into visualizations remains difficult due to its distinct paradigms and inherent performance challenges. We identify three classes of video data visualization – synchronization, annotation, and transformation – and integrate them into the Vega declarative grammar. We show that these abstractions enable high-performance implementation. To reconcile Vega’s instantaneous dataflow with video player state, we introduce a split-signal architecture that preserves declarative semantics while masking video update delays. We detect continuous scrubbing interactions at compile time to apply encoding-aware optimizations that improve responsiveness by up to 4 × 4\times . We also repurpose VOD protocols to transform videos in real time, delivering sub-200ms updates even on multi-hour-long compilations. These contributions enable seamless integration of conventional and video data visualization. \vgtccategory Research \preprinttext Preprint \manuscriptnote \authorfooter Dominik Winecki and Arnab Nandi are with The Ohio State University. E-mail: {winecki.1 | nandi.9}@osu.edu. Introduction keywords Video data, visualization grammar, declarative visualization, interactive exploration. We consider interactive visualizations combining video data with conventional—non-video—data . Multimodal data pipelines collect significant volumes of associated metadata, tabular data, and time-series data, and generate derived datasets and computer vision model inference outputs to complement video data streams. Video data is no longer a stand-alone medium; regardless of the application, additional data is captured alongside it or extracted from it. Just as conventional data requires conventional visualization, hybrid conventional/video data requires multimodal conventional/video visualization. Creating data visualizations that incorporate video data remains challenging. Semantically, most modern tools build on the grammar of graphics [ 28 ] , which is conceptually incongruent with incorporating video player elements or transforming video data. Additionally, dataflow graph architectures for dynamic/interactive visualizations assume a consistent, unified state with instantaneous propagation. However, resynchronizing video playback state is relatively slow, quickly exceeding the strict latency constraints required for a good user experience. Video systems are inherently difficult due to the volume and computational requirements of videos. A single frame can contain more information than an entire data dashboard. Further, videos are encoded/compressed out of necessity, and only a small portion is kept in memory at a time. These factors are antithetical to the traditional principles of high-performance visualization. Hardware acceleration and decades of optimizations make high-resolution video playback possible, even on commodity devices. However, the video playback state is still subject to what is, in essence, inertia . A video in playback will continue to play without delay, but initially opening the video, seeking to a new position, or changing video content, i.e., interactions , still occur on the scale of hundreds of milliseconds to multiple seconds. Such inertial resistance to interaction creates detrimental lag across the entire visualization when a unified approach to visualization/playback state is used. These challenges arise against a backdrop of rapid growth in video data driven by the proliferation of high-resolution cameras, declining storage and compute costs, and unprecedented advances in machine learning models’ visual understanding. Cameras have become the primary way to observe the physical world — the ultimate sensor — making video a quintessential part of multimodal data pipelines. In many of these applications, humans are uniquely capable of integrating subtle spatial, temporal, and contextual cues in video with structured summaries in charts or tables. It is therefore critical that they be equipped with useful video data visualization tools that leverage humans’ unmatched visual reasoning across both modalities. Consider a researcher studying vehicular near-misses and aggressive driving events in a fleet equipped with dash-cams and telemetry monitors. Per-event visualization may involve synchronizing time-series telemetry plots with video feed(s), charting trajectories on a map, annotating the videos with lane markers or object detections (e.g., cars, people, bikes), and displaying event metadata. Interactions would be playing the video and jumping to interesting positions on the timeline. Beyond single-event visualization, cross-event visualization covers the full dataset. Each event corresponds to a tuple, and correlations and aggregations can be plotted; subsequent interactions can filter the events. While this is a typical exploration approach, adding video-native elements can provide further value. A video compilation can display all events and be linked to histograms or scatter plots, with brushing triggering real-time filtering of the video compilation. Further, compilations can be sorted by an attribute, such as event time-of-day, so users can use the video timeline progress bar to quickly visualize how different percentiles of the attribute appear in the videos. For example, the start of the compilation corresponds to early-morning events, the middle to midday events, and the end to evening events. This linking can also be highly integrated and bi-directional. A scatter plot, for example, can highlight the currently displayed video event, and clicking on a point in the scatter plot can jump the video to that event. Visualizations that can edit and transform videos — beyond just syncing and annotating them — are clearly powerful exploration paradigms, but are either impractical or impossible under current systems. The preceding motivating example spans from long-established techniques to novel ones. We split these into three categories: syncing, annotating, and linked video transformation. Syncing with video is well established, with numerous visualization systems supporting it, yet they lack expressivity and portability; many offer only temporal synchronization with line plots, with limited potential for further development, interface exploration, or performance optimization. Conversely, widely used visualization grammars excel at composing a wide range of graphics linked with interactions. Annotation is conceptually similar and is relegated to domain-specific tasks, particularly drawing bounding boxes or labels. Entirely unexplored is linked video transformation, in which interactions with a conventional part of a visualization update the content of a video player, not just its playback state. This presents a challenge both in the semantics of representing these visualizations and in significant performance hurdles to ensuring low-latency real-time updates. The closest approach in use is to provide links or embedded players for individual source video files, a primitive approach comparable to displaying a text field. Few visualization systems support either of the first two video data visualization classes; none support the latter, and none tie these ideas together. In this paper, we build an extension to Vega [ 24 ] to support first-class video data visualization. We extend its grammar and that of higher-level systems, such as Vega-Lite [ 23 ] , to support declarative integration of video player elements. We provide a unified representation of time, making specifications indistinguishable from the existing dataflow paradigm while masking the underlying synchronization control loop and providing semantically correct behavior. Further, the slowest synchronization-based updates are those that seek videos, so we build on our time handling to classify them and select optimal seeking strategies, reducing user latency during interactions that trigger seeking. We introduce a video transformation layer that supports real-time video updates via declarative data transformations, performing lightweight editing. We release our implementation, vega-video.js, as open source software. Contributions In this paper, we extend the Vega grammar to support first-class video data visualization and resolve the performance challenges necessary for interactive use. Specifically, we contribute: • A declarative grammar that extends Vega to support video players, frame-level annotations,

Read the complete text