BasketLiDAR: The First LiDAR-Camera Multimodal Dataset for Professional Basketball MOT

Ryunosuke Hayashi, Kohei Torimi, Rokuto Nagata, Kazuma Ikeda, Ozora Sako, Taichi Nakamura, Masaki Tani, Yoshimitsu Aoki, Kentaro Yoshioka · 2025

Read BasketLiDAR: The First LiDAR-Camera Multimodal Dataset for Professional Basketball MOT by Ryunosuke Hayashi, Kohei Torimi, Rokuto Nagata, Kazuma Ikeda, Ozora Sako, Taichi Nakamura, Masaki Tani, Yoshimitsu Aoki, Kentaro Yoshioka (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15299.txt.

Introduction

Ryunosuke Hayashi and co-authors build BasketLiDAR, a data set for tracking basketball players in three dimensions. Their 2025 paper joins LiDAR point clouds with synced camera views from professional games. Player tracking feeds tactic study and fan experience, but video alone is flat and needs costly reconstruction to recover depth.

Basketball is a hard case: ten players move fast in a small court, with frequent blocking from body contact. The data set holds four thousand four hundred forty-five frames and over three thousand player identities, matched across three LiDAR sensors and three cameras. Games include five-on-five and three-on-three play by professionals. A new tracking framework built on the data lifts accuracy while cutting computing cost.

Catalog record

CREATORRyunosuke Hayashi, Kohei Torimi, Rokuto Nagata, Kazuma Ikeda, Ozora Sako, Taichi Nakamura, Masaki Tani, Yoshimitsu Aoki, Kentaro Yoshioka
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15299

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BasketLiDAR: The First LiDAR-Camera Multimodal Dataset for Professional Basketball MOT Ryunosuke Hayashi, Kohei Torimi, Rokuto Nagata, Kazuma Ikeda, Ozora Sako, Taichi Nakamura, Masaki Tani, Yoshimitsu Aoki, Kentaro Yoshioka Abstract . Real-time 3D trajectory player tracking in sports plays a crucial role in tactical analysis, performance evaluation, and enhancing spectator experience. Traditional systems rely on multi-camera setups, but are constrained by the inherently two-dimensional nature of video data and the need for complex 3D reconstruction processing, making real-time analysis challenging. Basketball, in particular, represents one of the most difficult scenarios in the MOT field, as ten players move rapidly and complexly within a confined court space, with frequent occlusions caused by intense physical contact. To address these challenges, this paper constructs BasketLiDAR, the first multimodal dataset in the sports MOT field that combines LiDAR point clouds with synchronized multi-view camera footage in a professional basketball environment, and proposes a novel MOT framework that simultaneously achieves improved tracking accuracy and reduced computational cost. The BasketLiDAR dataset contains a total of 4,445 frames and 3,105 player IDs, with fully synchronized IDs between three LiDAR sensors and three multi-view cameras. We recorded 5-on-5 and 3-on-3 game data from actual professional basketball players, providing complete 3D positional information and ID annotations for each player. Based on this dataset, we developed a novel MOT algorithm that leverages LiDAR’s high-precision 3D spatial information. The proposed method consists of a real-time tracking pipeline using LiDAR alone and a multimodal tracking pipeline that fuses LiDAR and camera data. Experimental results demonstrate that our approach achieves real-time operation, which was difficult with conventional camera-only methods, while achieving superior tracking performance even under occlusion conditions. The dataset is available upon request at: https://sites.google.com/keio.jp/keio-csg/projects/basket-lidar Figure 1. (a) BasketLiDAR captures comprehensive court coverage without blind spots using three sets of calibrated camera-LiDAR sensor systems. (b) BasketLiDAR data: LiDAR point clouds, multi-view camera images, and player bounding boxes in each point cloud with fully synchronized IDs across all modalities. 1. Introduction Player trajectory tracking in sports analysis plays a central role in tactical analysis, performance evaluation, and enhancing spectator experience. Traditionally, coaches and analysts have relied on manual video analysis, but recent advances in computer vision technology have brought attention to automated approaches using Multiple Object Tracking (MOT). In particular, MOT, which performs player position estimation and ID association over time, has become an essential technological foundation for real-time tactical analysis, detailed player statistics generation, and immersive spectator experiences utilizing AR technology. Traditional sports 3D MOT systems have primarily adopted multi-view camera approaches ( Wu et al. 2020 ) . These systems have employed MOT integrating multiple camera feeds to obtain accurate player position information and movement patterns necessary for tactical analysis. However, since camera footage is inherently two-dimensional information, acquiring the 3D spatial information necessary for sports analysis requires multi-stage processing including person detection in each camera view, feature extraction, inter-camera correspondence, and 3D reconstruction. This complex processing pipeline incurs significant computational costs, creating major constraints for deployment in sports environments that require real-time performance. This represents a critical challenge that hinders the proliferation of data-driven sports analysis and value delivery to players, coaches, and spectators. Therefore, this study poses the following research question: Can introducing LiDAR sensors simultaneously achieve improved accuracy and reduced computational cost in sports MOT? LiDAR sensors are 3D sensors with proven track records in person and vehicle detection in autonomous driving, possessing promising characteristics for solving sports MOT challenges. First, LiDAR can directly acquire high-precision 3D point clouds of surrounding environments at 10 FPS, eliminating the complex processing required to estimate 3D space from 2D information like camera footage. Second, since point cloud data itself has spatial structure, multi-sensor data integration can be achieved through extremely simple processing, eliminating the need for complex 3D reconstruction processing, likely enabling reduced computational cost for the entire system and real-time operation. To validate this hypothesis, we selected basketball as our subject, where occlusions occur frequently and 3D analysis is particularly challenging. Basketball represents one of the most difficult scenarios in sports MOT, as ten players move rapidly and complexly within a confined court (28m×15m), with frequent occlusions caused by intense physical contact. This paper makes the following contributions: (1) LiDAR-Camera multimodal sports MOT dataset: We construct BasketLiDAR , the first synchronized MOT dataset combining LiDAR point clouds and multi-view camera footage targeting professional basketball players, with ID and 3D position annotations for all players. (2) Proposal of fusion-based MOT framework: We construct an multimodal MOT framework integrating LiDAR-based tracking with ID re-identification from camera footage, achieving real-time and high-precision MOT performance. (3) Research infrastructure for sports MOT: We provide open-source publication of the dataset and the model implementation, contributing to research advancement in the LiDAR-based sports analysis field. 2. Related Works 2.1. Sports MOT Datasets Current sports MOT datasets are predominantly based on RGB videos. Representative examples include SportsMOT ( Cui et al. 2023 ) , which is an MOT benchmark that annotates bounding boxes and IDs for all players on 240 monocular RGB videos (approximately 150,000 frames) containing indoor side-view footage similar to NBA broadcast angles. TeamTrack ( Scott et al. 2024 ) is a large-scale benchmark using 4K-8K high-resolution fixed camera footage captured from diverse viewpoints including fisheye side views and drone overhead perspectives. TrackID3x3 ( Yamada et al. 2025 ) is a dataset for integrated tracking and pose estimation evaluation that provides annotations including bounding boxes, IDs, and 10-joint 2D skeletal points for fixed camera and drone footage. In the autonomous driving field, numerous multimodal datasets combining LiDAR and cameras have been constructed, including KITTI ( Geiger et al. 2012 ) , Waymo ( Sun et al. 2020 ) , and nuScenes ( Caesar et al. 2020 ) . However, fundamental differences exist between autonomous driving and sports scenes in terms of tracking targets (vehicles vs. humans), motion patterns (predictable traffic rules vs. complex and unpredictable sports movements), and sensor placement (vehicle-mounted vs. fixed installation). Therefore, direct application of autonomous driving datasets to sports MOT is challenging, making the construction of specialized datasets addressing sports-specific challenges essential. To summarize, all existing sports MOT datasets rely solely on RGB footage, with 3D positional information limited to indirect estimation from 2D imagery. In contrast, BasketLiDAR constructed in this study is the first multimodal dataset in the sports 3D MOT field with fully synchronized LiDAR point clouds and multi-view camera footage. Through direct 3D spatial measurement by LiDAR, it provides a novel research foundation enabling high-precision and real-time 3D MOT. 2.2. Camera-based 3D MOT Technology Most MOT approaches for single cameras are implemented using Tracking-by-Detection (TBD), which has evolved by associating frame-by-frame detection results from object detectors with short-term state prediction using motion models and appearance information using Re-ID features ( Han et al. 2019 ; He et al. 2023 ) . SORT and DeepSORT ( Bewley et al. 2016 ; Wojke et al. 2017 ) introduced appearance-based similarity using Re-ID features into the SORT framework, suppressing ID switches in occlusion and high-density scenes. Recently, ByteTrack ( Zhang et al. 2022 ) proposed a novel algorithm that associates previously discarded low-confidence detection results with existing tracks, significantly improving true positives for not losing targets under occlusion. However, extending these single camera-based

Read the complete text