Velocity and stroke rate reconstruction of canoe sprint team boats based on panned and zoomed video recordings Julian Ziegler 1,∗, Daniel Matthes 1,∗, Finn Gerdts 1, Patrick Frenzel 1, Torsten Warnke 2, Matthias Englert 2, Tina Kövari 3, Mirco Fuchs 1 Abstract Pacing strategies, defined by velocity and stroke rate profiles, are essential for peak performance in canoe sprint. While GPS is the gold standard for analysis, its limited availability necessitates automated video-based solutions. This paper presents an extended framework for reconstructing performance metrics from panned and zoomed video recordings across all sprint disciplines (K1–K4, C1–C2) and distances (200m–500m). Our method utilizes YOLOv8 for buoy and athlete detection, leveraging the known buoy grid to estimate homographies. We generalized the estimation of the boat position by means of learning a boat-specific athlete offset using a U-net based boat tip calibration. Further, we implement a robust tracking scheme using optical flow to adapt to multi-athlete boat types. Finally, we introduce methods to extract stroke rate information from either pose estimations or the athlete bounding boxes themselves. Evaluation against GPS data from elite competitions yields a velocity RRMSE of 0.020±0.011 ( ρ = 0.956 \rho=0.956 ) and a stroke rate RRMSE of 0.022±0.024 ( ρ = 0.932 \rho=0.932 ). The methods provide coaches with highly accurate, automated feedback without requiring on-boat sensors or manual annotation. † † institute: Laboratory for Biosignal Processing, Leipzig University of Applied Sciences, Leipzig, Germany ∗ Equal Contributions and Research Group Canoeing, Institute for Applied Training Science (IAT), Leipzig, Germany and German Canoe Federation, Duisburg, Germany 1 Introduction Pacing strategies are fundamental to achieve top performance in canoe sprint. An elite performance is characterized by the complex interplay between velocity profiles and stroke rate profiles, which indicate how velocity and stroke rate, respectively, evolve over the course of the race. To gain a competitive advantage, coaches and sports scientists require precise, high-frequency data to analyse tactical execution of all competitors throughout a race. While Global Positioning System (GPS) technology is currently considered the gold standard for acquiring such data, its practical utility is often constrained. In high-stakes competition, GPS data is rarely available for all participants, and the deployment of on-boat sensors can be logistically cumbersome or prohibited by regulations. As an alternative, video-based analysis offers a non-invasive means of data collection. However, manual video analysis using panned and zoomed recordings from the shore is frequently deemed unfeasible. The skewed perspective inherent in side-view filming renders the manual reconstruction of absolute boat positions extremely difficult and prohibitively labour-intensive. Consequently, there is a significant need for automated systems capable of transforming standard broadcast or coach-recorded footage into accurate kinematic data. In previous work, Matthes et al. introduced a method to automatically reconstruct velocity profiles for single-person boats (K1/C1) using a fixed camera position 7 . This approach utilized a YOLOv8 14 object detection model to identify buoys and athletes, leveraging the known geometry of the buoy grid to estimate homographies and establishing buoy correspondences between frames via optical flow (cmp. Fig. 1 ). While this method proved highly accurate for 500m single-person races, it was limited in scope to individual disciplines and did not provide means for stroke rate reconstruction. Figure 1: Visualization of the reconstructed scene geometry for a K1 canoe race over the distance of 500 m according to 7 . Orange lines correspond to lane boundaries, blue ones to boundaries between equidistant segment, and yellowish dots where orange and blue lines cross are expected buoy location alongside the track. See Sec. 3 for an outline of the scene reconstruction method. Numbers in red coloured annotations at boat tips correspond to the distance travelled of each boat, which can further be utilized to reconstruct velocity profiles. Best viewed on screen. This paper significantly extends that framework in three key areas, and expands the methodology to team boats (K2, C2, K4). We first introduce an automated calibration to determine the athlete offset used to localize the boat on a per video, per lane basis. To achieve this, a U-Net model is trained to detect boat tips. The multi-athlete classes introduce significant complexity due to potential detection failures among multiple athletes. To resolve this, we implement a robust tracking logic that identifies frames with complete detections and utilizes optical flow to maintain athlete order, as well as position estimates in adjacent frames. Third, we incorporate stroke rate extraction into the pipeline. Two methods of capturing a one-dimensional motion signal are introduced, either using a ViTPose 16 model or the YOLO bounding boxes. From the motion signal, we derive stroke rate. A comprehensive evaluation of the system across a diverse range of distances (200m, 500m) and boat types is provided. The resulting framework delivers a robust, automated, camera-based system for tactical race analysis, providing immediate, actionable feedback across all canoe sprint disciplines without the need for on-boat hardware. 2 Related Works Exploiting static scene geometry to establish correspondences between image observations and real-world coordinates is a well-established strategy in computer vision. When parts of the environment provide reliable geometric structure (e.g., planes, lines, or repeated landmarks), these constraints can be used to infer camera parameters and spatial mappings with reduced dependence on manual annotation or specialised sensors. Such approaches have been applied successfully to problems including camera calibration, object localisation, and scene understanding 9 ; 12 . Sport analytics is a particularly strong application domain for geometry-based correspondence, because the environment often contains fixed markings and largely standardised camera viewpoints. In field sports, line markings and known layout dimensions enable estimation of homographies or full camera models, which in turn support metric player tracking and the extraction of spatiotemporal performance measures from broadcast or consumer-grade footage 17 ; 5 ; 3 ; 18 . By anchoring visual measurements to a calibrated world coordinate frame, these methods provide scalable alternatives to labour-intensive per-frame annotation and facilitate robust analysis under realistic capture conditions. In competitive canoeing and related water sports, recent work has begun to adapt these principles to aquatic environments, where reflections, spray, and limited stable reference structure make calibration and measurement more challenging. Prior studies have leveraged waterline segmentation as a persistent visual reference for technique assessment 15 , and demonstrated the feasibility of video-based metric extraction for performance evaluation 2 . Beyond 2D analysis, monocular reconstruction of the full 3D paddle motion has also been explored 8 , indicating that meaningful kinematic descriptors can be obtained even without multi-camera setups. These developments align naturally with advances in temporal 3D pose estimation 10 and learning-based camera calibration pipelines 11 , which together reduce reliance on controlled setups and enable increasingly automated analysis. Complementary lines of research address higher-level race and crew dynamics, such as the quantification of stroke synchronisation 13 , the analysis of pacing profiles 4 , and broader algorithm-based approaches for competitive canoeing performance analysis 1 . 3 Previous Work Figure 2: Shown is the scene geometry of a regatta course (left) and an approximate orthogonal camera view on the scene (right). For clarity, yellow buoys highlighted in the scene are those visible in the right image. Corresponding buoys in both views can be identified based on their lane boundaries B_{i} and track segments S_{j} . Correspondences are used to estimate a homography and propagated to adjacent frames. Adapted from 7 . This work builds on earlier methods for reconstructing boat velocity profiles from race video 7 . The original approach exploits the standardised geometry of a regatta course to turn image measurements into metric positions on the planar water surface (Fig. 2 ). Buoys and athletes are detected in each frame using a fine-tuned YOLOv8 model, and the fixed buoy layout provides reliable reference points to estimate an image-to-course mapping. Because correspondences are easiest to initialise in a near-orthogonal view, the mapping is first set up in a frame where lane boundaries appear approximately horizontal, fitted robustly (RANSAC), and then carried through the full sequence by tracking buoy locations with Lucas-Kanade optical flow. Propagating the mapping forward to the finish and backward to the start yields a consistent track coordinate frame for every frame, even when buoy detections are intermittent due to occlusions or missed detections. Boat position is approximated indirectly from the detected athlete location by using a point near the bottom of the athlete bounding box as a proxy for where the boat lies in the image. The resulting trajectory is smoothed to reduce jitter from detection noise and athlete motion, velocity is obtained by differentiating the smoothed position over time, and the final profiles are reported as averages over fixed 12.5 m segments in line with common coaching practice. Despite its effectiveness for single-athlete boats, the method has important limitations that motivate the present work. Most notably, it does not extend cleanly to crew boats, because multiple athletes are visible at once and often overlap; the detection that serves as a reliable proxy in a single-athlete setting becomes ambiguous when several athletes could plausibly define the boat position. In addition, the boat tip is not observed directly and is instead approximated by adding a constant empirical offset (2.4 m, tuned to German boat measurements) to the mapped athlete position, which can introduce a systematic bias when boat class, athlete placement, or recording conditions differ. This issue is amplified for team boats, where the correct offset is no longer global: it depends on which seat (and thus which athlete) is used as the proxy, making a single correction insufficient. Finally, the pipeline focuses on velocity profiles and does not provide stroke profile information. 4 Methodology In this section, we present our proposed methodology. The framework is designed specifically to address the limitations identified in prior work, particularly regarding multi-athlete boat types. The primary challenge in these environments is the increased complexity of spatial association and the necessity for high-precision geometric calibration that accounts for the unique configuration of team boats. To overcome these hurdles, we first introduce the general mathematical framework and adjust notations to also handle team boats. Second, we present an extended dataset, which includes the multi-athlete boat types K2, C2, K4, as well as more C1 and K1 videos than the original dataset. Moreover, a boat tip dataset to train the U-Net was also curated. We then detail our automatic boat tip calibration, enabling superior boat localization compared to using a static, empirical offset. To this end, a U-Net model was implemented to detect the boat tip given an image patch, which we feature in section 4.4 . Further, we provide a description of our multi-athlete tracking using optical flow, enabling athlete order disambiguation and positional tracking in the event of failed detections or occlusions. 4.1 Overview First, we expand the method to handle multi-person boats and update the notation accordingly. Consistent with previous work, we estimate a homography \mathbf{H}_{i} for each frame f_{i} , which enables the mapping of image points on the water surface plane to two-dimensional real-world coordinates. We utilize a trained YOLOv8 model to detect athletes in every frame. The set of detections per frame is denoted as \mathcal{B}_{i}^{\prime} , where the prime symbol ( ′ ) indicates vectors or sets residing in the image space. The 2d image position of an athlete \mathbf{a}^{\prime} is defined as the centre of the bottom edge of their respective bounding box \mathbf{b}^{\prime}\in\mathcal{B}_{i}^{\prime} , resulting in the set of athlete image positions \mathcal{A}^{\prime}_{i} . Given the following transformation: w\begin{pmatrix}\mathbf{a}\\ 1\end{pmatrix}=\mathbf{H}_{i}\cdot\begin{pmatrix}\mathbf{a}^{\prime}\\ 1\end{pmatrix} (1) we obtain the 2d positions of each athlete in world coordinates (i.e., a position on the water surface) and therefore a set of athlete world positions \mathcal{A}_{i} per frame, where w is the homogeneous scaling factor. By incorporating knowledge of lane boundaries, we can partition these detections into lane-specific subsets \mathcal{A}_{i,j} , where j represents the lane number. For a given lane j and frame f_{i} , the set \mathcal{A}_{i,j} contains N elements, depending on the boat type (e.g., K1, K2, or K4). Each individual athlete position is denoted as \mathbf{a}_{i,j,k} , with k\in\{1,\dots,N\} . These positions are ordered according to the boat’s direction of travel, i.e. the first athlete is the one closest to the tip of the boat. Because the position of a boat in canoe sprint races is defined by its tip, we apply a set of correction offsets \mathbf{o}_{j,k} from each athlete k to the boat tip. The estimation of these offsets is detailed in Section 4.3 . Finally, the boat position \mathbf{x}_{i,j} in frame f_{i} and lane j in real-world coordinates is computed as: \mathbf{x}_{i,j}=\frac{1}{N}\sum_{k=1}^{N}(\mathbf{a}_{i,j,k}+\mathbf{o}_{j,k}) (2) Evaluating these coordinates over time t yields a lane-specific distance-time signal \mathbf{x}_{j}(t) , the further application of which is discussed in the subsequent sections. Figure 3: Visualization of all boat classes in our main dataset. C refers to canoe, where single-bladed paddles are used. K refers to kayak, where double-bladed paddles are mandated. The number indicated the number of athletes per boat. 4.2 Datasets This section presents the datasets used throughout this study. We distinguish between a main dataset and a curated dataset that is only used to train the boat tip detection model. Both are datasets are outlined below. The main dataset comprises 5 boat types, single-seated C1 and K1, double seated C2 and K2, and quadruple seated K4, where C refers to a canoe (single bladed paddle) and K to a kayak (double bladed paddle). See figure 3 for a sample image of each boat type. The video recordings were obtained from the German National Qualifiers 2023 in Duisburg, the 2023 ICF Canoe Sprint World Championships in Duisburg, and the 2024 ICF Canoe Sprint World Cup in Poznan, Poland. We captured at 4K resolution and 25 fps from an elevated position 40-90 m before the finish line. GPS data was acquired by GPS modules mounted at the stern, and data post-processed to align trajectories to a common starting reference. Gyroscopic stroke rate data is processed in the same way. Overall, the main dataset contains 54 GPS and Gyro tracks across 40 race video recordings, as some videos include multiple boats for which GPS and Gyro data is available. We include both 200 m and 500 m races. Table 1 shows the exact split across all boat types and race distances. Table 1: Contents of our main dataset. Distance K1 C1 K2 C2 K4 Total 200 m 8 3 – – – 11 500 m 21 6 7 5 4 43 Total 29 9 7 5 4 54 Furthermore, we build a boat tip dataset of image patches with a 150\times 150 pixel resolution. The patches were selected according to the method described in Sec. 4.3 , resulting in patches where the boat tip is near the centre coordinate of the patch. In each patch, the boat tip is manually annotated. The resulting dataset comprises 550 train samples, 124 validation datapoints and 98 testcases. 4.3 Automatic Boat Tip Offset Calibration In this section, we detail the derivation of the correction offsets \mathbf{o}_{j,k} , which are applied to athlete positions to localize the boat. While the previous work focused on single-seater vessels using a single empirical offset, we now introduce a method to automatically calibrate all N offsets for a boat in lane j . Figure 4: Graphical Illustration of Automatic Boat Tip Offset Calibration. (1) For every detected athlete bounding box (green), we assume the centre of the bottom edge to be the image position of that athlete (blue point). (2) Athlete positions are transformed to the 2d world space, and a standard offset is applied to the mean position of the athletes (orange). (3) This position is reprojected into the image, and defined as centre of ROI. (4) ROI is input into a trained U-Net, which predicts the exact tip location (bright green). (5) Exact tip location and athlete positions in the image are again transformed to world coordinates. (6) Individual athlete offsets are calculated (red arrows). Best viewed on screen. To achieve this, a sequence of five frames is selected from a near-perpendicular view where all athletes are detected. For each frame i , we follow the pipeline illustrated in Fig. 4 . First, an initial estimate of the boat position in world coordinates, \tilde{\mathbf{p}}_{i,j} , is calculated by applying a boat-type specific empirical offset \mathbf{o}(N) to the mean of the athlete positions \mathcal{A}_{i,j} : \tilde{\mathbf{p}}_{i,j}=\mathbf{o}(N)+\frac{1}{N}\sum_{k=1}^{N}\mathbf{a}_{i,j,k}. (3) The corresponding image position \tilde{\mathbf{p}}^{\prime}_{i,j} serves as the centre pixel for a 150\times 150 pixel Region of Interest (ROI), i.e. a patch that is expected to contain the true boat tip. This ROI is processed by a U-Net model specifically trained for boat tip detection (see Sec. 4.4 ). The model outputs the precise image location of the boat tip, \mathbf{p}^{\prime}_{i,j} , which is then transformed back into world coordinates \mathbf{p}_{i,j} using the homography matrix \mathbf{H}_{i} . Consequently, the individual athlete offsets are derived as: \mathbf{o}_{j,k}=\mathbf{p}_{i,j}-\mathbf{a}_{i,j,k}. (4) To ensure temporal robustness and mitigate detection noise, these offsets are averaged over the five-frame sequence to produce the final correction offsets. 4.4 U-Net model for Boat Tip Detection We aim to learn a U-Net model to predict the boat tip present in a 150\times 150 pixel patch, i.e. regressing the position of the tip. For this, we formulate this problem as a classification of all pixels to either be or not be the boat tip. The target of the U-Net is a heatmap built from the ground truth boat tip position and a Gaussian kernel with a specified \sigma . This approach is common in other key point detection tasks such as pose estimation 16 or sports field registration 5 .The principle is visualized in Fig. 5 . Figure 5: Principle of boat tip localization using a U-Net. The image annotation (green dot) of the boat tip (left) is encoded via a heatmap with gaussian kernel (middle). The U-Net is trained to predict this heatmap (right) from which the predicted position (red cross) is derived using argmax over the predicted heatmap. 4.4.1 Model Architecture The neural network utilizes a ResNet-18 backbone, pretrained on ImageNet, as its encoder. This stage is responsible for generating deep, low-resolution feature maps that identify the existence and approximate spatial placement of keypoints. To maintain the integrity of the spatial structure during feature extraction, Groupnorm is implemented. The custom decoder restores spatial resolution through successive upsampling stages. A critical component of this architecture is the use of skip connections, which transfer low-level feature information directly from the encoder to the decoder. This mechanism facilitates a significantly more precise localization of the features identified in the earlier layers. The decoder consists of convolutional layers with ReLU activation and concludes with an output head that generates a heatmap of dimensions (1,H,W) . 4.4.2 Training and Optimization Training is conducted using a dual-loss strategy to maximize localization accuracy. Binary Cross Entropy (BCE) loss is applied to the heatmaps to identify probable pixel locations, while Mean Squared Error (MSE) loss is utilized on the final coordinates to further refine the predictions. The optimization process employs a Onecycle scheduler, with the decoder’s learning rate set ten times higher than that of the encoder. Furthermore, a sigma curriculum is applied to the dataset heatmaps, where the value of \sigma is reduced from 3 to 1.5 throughout the training duration. This curriculum forces the model to achieve increasingly accurate localization as the training progresses. 4.5 Robust Multi-Athlete Boat Localization Until now, we have only considered frames in which the number of detected athletes per lane, i.e. |\mathcal{A}_{i,j}| , equals the number of athletes N in the respective boat type. This, however, is of course not always the case. Furthermore, if any of the N athletes is failed to be detected, not only is that athlete’s position unknown, it makes it impossible to infer the correct positional order of all other athletes. This, in turn, results in the correction offsets to not be able to be applied. Given the set of all frames \mathcal{F} , there exist lane specific subsets \mathcal{F}^{c}_{j} where all athletes of that lane are correctly detected in the frames within this subset. Without further handling, the distance-time-signal \textbf{x}_{j}\left(t\right) would not be complete, as only frames f_{i}\in\mathcal{F}^{c}_{j} could be used to infer the boats’ position. In the following, we present multiple solutions. 4.5.1 Linear Interpolation The first, most simple solution is to linearly interpolate all boat positions \mathbf{x}_{i,j} where f_{i}\notin\mathcal{F}^{c}_{j} . Interpolation for frame f_{i} is based on two frames f_{l} and f_{m} , f_{l},f_{m}\in\mathcal{F}^{c}_{j} , where l