A Modular Pipeline for 3D Object Tracking Using RGB Cameras
Lars Bredereke, Yale Hartmann, Tanja Schultz · 2025
Read A Modular Pipeline for 3D Object Tracking Using RGB Cameras by Lars Bredereke, Yale Hartmann, Tanja Schultz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04322.txt.
Introduction
Lars Bredereke, Yale Hartmann, and Tanja Schultz build a modular system that tracks many moving objects in three dimensions using ordinary webcams, in a 2025 paper. Most trackers flatten motion onto a plane or follow a single object. Their pipeline instead combines several fixed, time-synced color cameras and reconstructs full 3D paths for every object in view. It was built for a dataset of people setting a table, filmed by six webcams amid millions of frames.
The pipeline must find small tableware items, work out where each camera sits, tell apart objects that nearly touch, and cope when items vanish behind hands. It spins up a fresh Kalman filter whenever an object appears and reports a confidence spread with each position. Across hundreds of table-setting trials it produced accurate paths with very little hand labeling, even when camera poses were unknown beforehand.
Catalog record
| CREATOR | Lars Bredereke, Yale Hartmann, Tanja Schultz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-04322
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Modular Pipeline for 3D Object Tracking Using RGB Cameras Lars Bredereke, Yale Hartmann, Tanja Schultz Abstract Object tracking is a key challenge of computer vision with various applications that all require different architectures. Most tracking systems have limitations such as constraining all movement to a 2D plane and they often track only one object. In this paper, we present a new modular pipeline that calculates 3D trajectories of multiple objects. It is adaptable to various settings where multiple time-synced and stationary cameras record moving objects, using off the shelf webcams. Our pipeline was tested on the Table Setting Dataset, where participants are recorded with various sensors as they set a table with tableware objects. We need to track these manipulated objects, using 6 rgb webcams. Challenges include: Detecting small objects in 9.874.699 9.874.699 camera frames, determining camera poses, discriminating between nearby and overlapping objects, temporary occlusions, and finally calculating a 3D trajectory using the right subset of an average of 11.12.456 11.12.456 pixel coordinates per 3-minute trial. We implement a robust pipeline that results in accurate trajectories with covariance of x,y,z-position as a confidence metric. It deals dynamically with appearing and disappearing objects, instantiating new Extended Kalman Filters. It scales to hundreds of table-setting trials with very little human annotation input, even with the camera poses of each trial unknown. The code is available at https://github.com/LarsBredereke/object_tracking . 1 Introduction Multiple Object Tracking (MOT) and Multi-Camera Multiple Object Tracking (MCMOT) are one of the most complex forms of object tracking. In our case, the information of 6 overlapping cameras’ recordings needs to be processed to calculate three-dimensional trajectories. Our implementation was tested on the Table Setting Dataset (TSD), but the modular nature of the pipeline makes it trivial to change the scenario to any other space recorded by time-synced, overlapping, and stationary cameras, where every tracked object should be visible to multiple cameras for most of the time. The pipeline can be adapted to a wide range of trackable objects, as YOLO-networks can detect a wide range of objects. The Table Setting Dataset provides multi-modal recordings for the activity of setting a table with tableware. To analyze table setting behavior, the human test subject is fitted with a motion capture suit, EEG, EMG, eyetracking and more. However, the manipulation of tableware objects is only recorded in video form. Therefore, actions performed by participants were previously only annotated manually, restricting the analysis of human behavior with machine learning. These manual annotations include timestamps, the hand used and the class of tableware object used. It does, however, not contain spacial information about where the object has been picked up and its trajectory. Extracting this spacial information of tableware objects from the provided webcam recordings shown in Figure 1 is the subject of this paper. Methods used include object detection using a YOLO-network, a gradient descent to optimize the camera’s position and orientation, and Kalman Filters to find the most likely trajectory of each object given the observations. Figure 1 : The TSD cameras used for this paper are referred to as back, ceiling, counter-top, front, table-side and table-top. 2 Related Research In computer vision, object tracking is a classic challenge with many applications for extracting trajectories from video streams. However, most object trackers focus on tracking in some two-dimensional space like the image plane or the ground plane, with comparatively few papers focusing on the full three degrees of freedom in movement. Alex Bewley et al Bewley et al. 2016 use a pipeline with a similar concept of a modular pipeline, where image detection algorithms predict bounding boxes, which are then processed with Kalman Filters. While our pipeline is built for post processing footage, this paper shows that such a pipeline can be easily developed into a real-time application. However, their approach only considers two-dimensional tracking. Cheng-Che Cheng et al Cheng et al. 2023 track multiple people on a 2D plane while dealing with temporary occlusions. Conceptually, they divide tracking into spatial and temporal association, allowing them to better deal with temporary occlusions. Similarly, Duy M. H. Nguyen et al Nguyen et al. 2022 track bounding boxes on a two-dimensional plane in three-dimensional space. Tracklets of individual cameras are then combined into trajectories. Moving to Multi-Camera Multi Object Tracking ( MCMOT ), Shuxiao Ding et al Ding et al. 2024 utilize cameras of a moving car to track objects in the street. However, they do not use explicit knowledge about camera poses to triangulate objects, as they use neural networks for this task. Moving back to single object tracking, Jingtong Li et al Li et al. 2020 track a drone from multiple cameras. They use 2D-tracklets of drones in image-coordinates to generate a 3D-trajectory, without the cameras being in sync. They also deal with cameras being in an unknown position. Noise in the trajectory is reduced by penalizing acceleration in 3D space. This paper does not fit our need for a fixed world-coordinate system, time-syncing and discriminating between identical looking objects. Jérôme Berclaz et al Berclaz et al. 2011 have a similar goal to us, as they track multiple objects in 3D space. However, they discretize 3D space and assume that two objects can not share the same position, which does not hold true for our use case, where the trajectories of objects can converge or diverge, as objects like plates are stackable. Finally, Fatih Cagatay Akyon et al Akyon et al. 2022 detect small objects in large images, which is a challenge that we also face with small tableware objects being recorded from across the room. However, the YOLO network we use can already detect small objects out of the box, even when objects are only a few pixels wide. 3 Data The Dataset used as input for our pipeline is the Everyday Activity in Science and Engineering (EASE) Table Setting Dataset (TSD) Mason et al. 2018 ; Mason et al. 2020 , which records humans setting a table. Dozens of sessions are provided, each containing around 6 trials. Each trial is a recording of a subject who is setting an initially empty table with tableware objects from a nearby counter. Depending on the trial instructions, the table is to be set for a varying number of guests with the type of meal also changing from trial to trial. One of the sensor modalities provided in the TSD are 6 cameras. They are stationary and record 720p at 30fps, with time-syncing provided by the Labstreaminglayer for Computational Neuroscience . There currently exist dozens of sessions with 6 trials lasting about 3 minutes each. An example for recorded images can be seen in Figure 1 , with the camera poses being shown in Figure 6 . The cameras are movable and can slightly change their position and orientation between trials. For some trials, individual cameras are not operational. In such cases, our pipeline works on only 5 cameras. 4 Proposed Pipeline Extracting 3D trajectories from multiple rgb cameras takes many steps which are shown in Figure 2 . This modular approach allows us to develop and optimize every step of the pipeline individually. For example, we can optimize our heuristics for Section 4.3 without having to rerun the YOLO network on millions