Vidformer: Drop-in Declarative Optimization for Rendering Video-Native Query Results
Dominik Winecki, Arnab Nandi · 2026
Read Vidformer: Drop-in Declarative Optimization for Rendering Video-Native Query Results by Dominik Winecki, Arnab Nandi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17221.txt.
Introduction
Dominik Winecki and Arnab Nandi built a system called Vidformer for work with video data. In video-native querying, users do not just get tables back. They get video back, with clips joined and data laid over frames. That last step, rendering, is the slow part. It is slower than the query work itself, and today it runs through slow scripts.
Vidformer adds fast, drop-in rendering to that flow. Users state what they want in a plain, declarative way. The system then speeds up how result videos are made. For teams that search large video sets, this cuts wait time. It makes hands-on video search feel quick and direct.
Catalog record
| CREATOR | Dominik Winecki, Arnab Nandi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-17221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vidformer: Drop-in Declarative Optimization for Rendering Video-Native Query Results Dominik Winecki, Arnab Nandi Abstract . When interactively exploring video data, video-native querying involves consuming query results as videos , including steps such as compilation of extracted video clips or data overlays. These video-native queries are bottlenecked by rendering , not the execution of the underlying queries. This rendering is currently performed using post-processing scripts that are often slow. This step poses a critical point of friction in interactive video data workloads: even short clips contain thousands of high-definition frames; conventional OpenCV /Python scripts must decode → \rightarrow transform → \rightarrow encode the entire data stream before a single pixel appears, leaving users waiting for many seconds, minutes, or hours. To address these issues, we present Vidformer , a drop-in rendering accelerator for video-native querying which, (i) transparently lifts existing visualization code into a declarative representation, (ii) transparently optimizes and parallelizes rendering, and (iii) instantly serves videos through a Video on Demand protocol with just-in-time segment rendering. We demonstrate that Vidformer cuts full-render time by 2–3 × \times across diverse annotation workloads, and, more critically, drops time-to-playback to 0.25–0.5 s. This represents a 400 × \times improvement that decouples clip length from first-frame playback latency, and unlocks the ability to perform interactive video-native querying with sub-second latencies. Furthermore, we show how our approach enables interactive video-native LLM-based conversational querying as well. 1. Introduction Figure 1. The naïve approach for rendering visualization videos is exceedingly slow. Vidformer uses a drop-in API shim to convert imperative code into declarative specifications and efficiently accelerate result rendering. Vidformer’s Video On Demand server then materializes segments just-in-time. Video data has become ubiquitous alongside tremendous advancements in computer vision and multimodal data processing in recent years. Domain-specific computer vision models now outperform humans using modest computational resources, while modern large language models (LLMs) and vision-language models (VLMs) are inherently multimodal, enabling zero-shot interactions with text, images, audio, and video. Additionally, video database management systems (VDBMSs) have made significant strides in running models over videos ( Kang et al. 2022 ; Kakkar et al. 2023 ) , largely solving inference at scale. These advancements have made limitations elsewhere in video data pipelines evident. We focus on one significant point of friction: rendering video/data visualization videos. In computer vision, annotating videos with model results is a critical technique for debugging and understanding models. Bounding boxes and object masks provide an intuitive understanding of the strengths and limitations of detection and segmentation models. Additionally, attention heatmaps allow visualizing the impact of individual features. Similarly, video databases are adept at answering queries over video datasets, such as which videos or frames match a search, and return relational/tabular results that can be used to create result compilation videos. Numerous other applications create similar video data visualizations, such as overlaying relevant telemetry information onto drone footage or synchronizing manufacturing sensor data with video streams from assembly lines. These use cases all share a similar goal: combining video streams with conventional data about them to create a new, enhanced visualization video. Such video data visualization workflows are widely used in scientific, commercial, and industrial applications. We have found that these varied tasks have nearly identical implementations: a script is written in a general-purpose programming language to perform the edit, creating a transformed visualization video. This approach offers unmatched interoperability with conventional data processing tools, along with the unbounded expressivity of Turing-complete scripts. However, this approach to rendering visualization videos is slow; even fast visualization scripts take many seconds or minutes to export a video. This significantly slows down iteration speed and incentivizes users to minimize their use of visualization videos and, when they do use them, to create only short videos. This poor experience is primarily a matter of tooling. One can run a conventional video editor, such as Adobe Premiere Pro or Apple Final Cut, on commodity hardware and watch edited videos in real time, demonstrating a clear tooling gap. In this paper, we detail a novel solution to instantly present script-defined video data visualizations to users. Accelerating the rendering of video data visualizations requires resolving the conflict between expressivity and the complexity of control flow. Simple tasks can be hard-coded into an interface to achieve reasonable performance, such as baking overlays into videos when they are first encoded. However, this approach cannot adapt to more complex tasks, as implementation complexity scales exponentially with the addition of functionality. Bespoke video data visualization scripts, such as those shown in Figure 2 , have excellent expressivity and interoperability with data, computer vision, and multimedia ecosystems. However, they fully render output videos and tightly integrate imperative control flow with unbounded complexity, precluding most opportunities for transparent optimization. Similarly, while it is possible to incorporate efficient algorithms into these scripts, this significantly increases their complexity. Even perfectly optimized, hardware-accelerated visualization scripts of this sort cannot reach the required speeds, as they fully render the result videos. Addressing these challenges is essential as video data workflows continue to proliferate. Video has emerged as a prominent data type across numerous scientific, commercial, and consumer applications. Despite this growth in video data volume, video has not experienced the same level of innovation in interactive interfaces as other data types, resulting in video data being slow to explore and analyze. Displaying video data visualizations quickly is a crucial step toward bridging this gap. In addition to the immediate benefit of accelerating existing workloads, a second-order effect is enabling new interaction paradigms. A long-standing chicken-and-egg problem is that novel features and applications are not developed because they would be too slow for practical use, and there is little incentive to improve their speed, as few features or applications exist to necessitate performance improvements. Solving the performance challenges of video data visualization allows novel experiences to be built on top of them. Namely, LLM-driven video-native querying, which we detail in Section 6.3 . This paper presents Vidformer , a novel approach for drop-in acceleration of video data visualization scripts. Our approach converts imperative video data visualization code into a declarative specification, then efficiently optimizes, parallelizes, and executes it to serve a transformed visualization video. We begin by detailing our method for lifting existing code into a declarative representation via symbolic computation ( Section 4 ). We implement an API compatibility layer for the most popular of these libraries, OpenCV’s Python library, enabling a one-line import vidformer.cv2 as cv2 patch to convert existing code. We then detail the design of our rendering engine ( Section 5 ), which executes declarative transformations on video, as DBMSs execute SQL to transform relational data. Finally, we propose an approach to use Video on Demand (VOD) streaming protocols to serve deferred video results ( Section 6 ). Short segments are lazily materialized just-in-time. From a user’s perspective, videos are instantly available; users can play and skip around the video, just as they did before. Most critically, we focus on creating a versatile solution; for our system to be effective, it does not just need to work in the typical case; it needs to work in every case, so we focus on worst-case video access patterns. By designing generalized solutions, our system is as expressive as general-purpose scripting languages