RocSync: Millisecond-Accurate Temporal Synchronization for Heterogeneous Camera Systems

Jaro Meyer, Frédéric Giraud, Joschua Wüthrich, Marc Pollefeys, Philipp Fürnstahl, Lilian Calvet · 2025

Read RocSync: Millisecond-Accurate Temporal Synchronization for Heterogeneous Camera Systems by Jaro Meyer, Frédéric Giraud, Joschua Wüthrich, Marc Pollefeys, Philipp Fürnstahl, Lilian Calvet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14948.txt.

Introduction

Jaro Meyer, Frédéric Giraud, and four co-authors tackle a practical filming problem: cameras of very different kinds rarely record in perfect sync. Their system, RocSync, aligns video streams to within a few milliseconds without relying on built-in hardware sync. It uses a custom LED device that the cameras can all see.

Mixed setups are the hard case, combining pro and consumer cameras, color and infrared sensors, and units with or without sound. RocSync is built to be cheap and general enough for field work where controlled capture is not possible. Good sync matters for three-dimensional reconstruction, pose tracking, and scene analysis, and the paper shows a workable route for real-world shoots.

Catalog record

CREATORJaro Meyer, Frédéric Giraud, Joschua Wüthrich, Marc Pollefeys, Philipp Fürnstahl, Lilian Calvet
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-14948

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RocSync: Millisecond-Accurate Temporal Synchronization for Heterogeneous Camera Systems Jaro Meyer, Frédéric Giraud, Joschua Wüthrich, Marc Pollefeys, Philipp Fürnstahl, Lilian Calvet Abstract Accurate spatiotemporal alignment of multi-view video streams is essential for a wide range of dynamic-scene applications such as multi-view 3D reconstruction, pose estimation, and scene understanding. However, synchronizing multiple cameras remains a significant challenge, especially in heterogeneous setups combining professional and consumer-grade devices, visible and infrared sensors, or systems with and without audio, where common hardware synchronization capabilities are often unavailable. This limitation is particularly evident in real-world environments, where controlled capture conditions are not feasible. In this work, we present a low-cost, general-purpose synchronization method that achieves millisecond-level temporal alignment across diverse camera systems while supporting both visible (RGB) and infrared (IR) modalities. The proposed solution employs a custom-built LED Clock that encodes time through red and infrared LEDs, allowing visual decoding of the exposure window (start and end times) from recorded frames for millisecond-level synchronization. We benchmark our method against hardware synchronization and achieve a residual error of 1.34 ms RMSE across multiple recordings. In further experiments, our method outperforms light-, audio-, and timecode-based synchronization approaches and directly improves downstream computer vision tasks, including multi-view pose estimation and 3D reconstruction. Finally, we validate the system in large-scale surgical recordings involving over 25 heterogeneous cameras spanning both IR and RGB modalities. This solution simplifies and streamlines the synchronization pipeline and expands access to advanced vision-based sensing in unconstrained environments, including industrial and clinical applications. A Preprint Keywords video synchronization; heterogeneous camera systems; sub-frame temporal alignment; multi-view 3D reconstruction; multi-view pose estimation; infrared and RGB imaging; computer vision 1 Introduction Temporal synchronization of multi-view video streams is essential for a wide range of computer vision tasks, including 3D reconstruction [ 1 , 2 , 3 ] , pose estimation [ 4 , 5 , 6 ] , and scene understanding [ 7 , 8 , 9 , 10 ] . The exact tolerance to synchronization errors depends heavily on the application. Semantic scene analysis tasks, such as surgical phase recognition, activity classification, or object detection, tend to aggregate information over extended temporal windows, making them relatively robust to minor temporal misalignments. In contrast, geometry-driven applications like multi-view 3D reconstruction and motion capture require that corresponding frames represent the same physical moment. In these cases, even single-digit millisecond offsets can cause significant reprojection errors, blurry artifacts in reconstructions, or inaccurate motion trajectories. In laboratory environments, specialized cameras with dedicated interfaces for hardware-based synchronization [ 11 , 12 , 13 ] can be used to ensure that the shutters are perfectly synchronized. However, such solutions are costly, require complex wired setups, and greatly restrict the range of compatible devices. In contrast, real-world deployments often involve camera arrays that integrate a heterogeneous mix of consumer and professional devices, many of which lack hardware synchronization capabilities. When hardware synchronization is not available, temporal alignment is often approximated by recording unsynchronized videos and then, during post-processing, estimating time offsets and pairing each frame from one camera with the closest frame from another stream. In practice, audio-based methods, such as clapping or other transient sound cues, remain common due to their simplicity and widespread availability of built-in microphones [ 14 , 15 , 16 ] . However, these methods are sensitive to environmental noise and cannot be used with devices lacking audio capture. Moreover, they constrain the spatial arrangement of the cameras, as all cameras must be within audible range and preferably positioned at similar distances from the sound source; otherwise, the relatively low propagation speed of sound causes arrival-time differences that can compromise synchronization. For example, in a room measuring 10 meters across, the sound of a clap will take roughly 30 milliseconds to travel from one side to the other, given the speed of sound in air ( \approx 343 m/s). This delay is already on the order of an entire video frame at 30 fps. One alternative solution is to use timecodes stored alongside the video [ 17 , 18 ] , but this requires external timecode systems or pre-synchronizing the cameras’ internal clocks before recording. More recently, content-based approaches have been proposed that estimate temporal relationships directly from image content, for example by tracking general scene features [ 19 ] , motion patterns [ 20 ] , lighting changes [ 21 ] , or human poses [ 22 ] across multiple cameras. While some of these methods achieve millisecond-level synchronization accuracy, they are typically tailored to narrow use cases and rely on specific video content, which limits their applicability in more diverse scenarios. Some systems address this by explicitly embedding visually encoded timecodes into the video [ 23 ] . However, existing approaches are constrained by the low and fixed refresh rates of consumer screens and are not suitable for scenarios involving infrared (IR) cameras. In contrast to hardware synchronization, post hoc alignment methods cannot ensure perfect frame correspondence, since the camera shutters operate independently. The residual error is bounded by half the frame period and thus scales with frame rate. For example, at 30 fps, the maximum misalignment can reach 16.7 ms, which is significant for fast motion. Although higher frame rates reduce this error, they also increase storage, power consumption (especially critical for battery-powered cameras), bandwidth, and processing costs, which can be prohibitive for long video sequences such as four-hour surgical recordings. It is therefore beneficial to estimate time offsets with sub-frame precision, enabling accurate frame interpolation for many downstream tasks. (a) (b) Figure 1: Our custom LED Clock . ( a ) Example of the device in use. ( b ) Rendering of the final PCB design with annotated features (red features encode the timestamp and blue features are used for detection & Euclidean rectification). The exposure window of the camera is visible in (a) as a red elliptical arc, which can be decoded with an accuracy of about 1 ms to enable sub-frame temporal synchronization. To address this gap, we propose a low-cost, portable, and camera-agnostic method for achieving millisecond-level temporal alignment across heterogeneous camera systems, ranging from inexpensive action cameras to high-end optical tracking systems. The proposed solution is designed to be scalable, supporting configurations ranging from as few as two cameras to several dozen, and remains robust in real-world scenarios. Central to our approach is a custom LED Clock , a 25\times 25\,\text{cm} device that visually encodes timestamps using LEDs (see Figure 1a ). An operator sequentially positions the LED Clock in front of each camera. Timestamps are encoded via two complementary LED components (Figure 1b ): a binary counter providing a global timestamp with 100 ms resolution, and a circular arrangement of LEDs illuminated sequentially at 1 ms intervals. Because camera sensors integrate light over a non-zero exposure time, the circular sequence appears in the image as an elliptical arc, whose start and end points indicate the beginning and end of the exposure window. For accurate spatial decoding, an ArUco marker [ 24 ] of known geometry is included at the center, enabling detection and homographic rectification of the captured LED components (i.e., binary counter and elliptical arc). Combining the binary timestamp with the arc-based exposure estimate yields a millisecond-accurate global timestamp for the

Read the complete text