Adding Another Dimension to Image-based Animal Detection

Vandita Shukla, Fabio Remondino, Benjamin Risse · 2026

Read Adding Another Dimension to Image-based Animal Detection by Vandita Shukla, Fabio Remondino, Benjamin Risse (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09210.txt.

Introduction

Shukla, Remondino, and Risse tackle depth loss when animals are filmed with one camera, where detection gives flat boxes with no facing hint. Their pipeline fits skinned multi-animal body models to estimate three-dimensional boxes, then projects them into image space as labels with a refined camera pose method. Cuboid face visibility metrics show which side of the animal the camera caught. Tests on the Animal3D set show accurate results across species and settings, a step toward monocular 3D animal detection.

Catalog record

CREATORVandita Shukla, Fabio Remondino, Benjamin Risse
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18947
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09210

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adding Another Dimension to Image-based Animal Detection Vandita Shukla, Fabio Remondino, Benjamin Risse Abstract Monocular imaging of animals inherently reduces 3D structures to 2D projections. Detection algorithms lead to 2D bounding boxes that lack information about animal’s orientation relative to the camera. To build 3D detection methods for RGB animal images, there is a lack of labeled datasets; such labeling processes require 3D input streams along with RGB data. We present a pipeline that utilizes Skinned Multi-Animal Linear models to estimate 3D bounding boxes and to project them as robust labels into 2D image space using a dedicated camera pose refinement algorithm. To assess which sides of the animal are captured, cuboid face visibility metrics are computed. These 3D bounding boxes and metrics form a crucial step toward developing and benchmarking future monocular 3D animal detection algorithms. We evaluate our method on the Animal3D dataset, demonstrating accurate performance across species and settings. 1 Introduction and related work Animal biometrics is a research field focused on quantifying phenotypical characteristics and their changes in individual animals 11 ; 6 . Visual animal biometrics (VAB) has emerged as a key technique for identifying and tracking animals based on unique physical features, with applications in conservation, livestock management, and anti-poaching 13 ; 12 . Recent advances in computer vision and machine learning have enabled non-invasive VAB, eliminating the need for tagging or microchipping 14 ; 1 . However, camera-based imaging, such as camera trap recordings, inherently reduces 3D objects to 2D projections. Standard object detection algorithms extract 2D bounding boxes, which help localize the animals in image-space but lack geometric context 19 . Image-based animal detection, without characterizing the visible orientation of animal profiles in the image, limits both the observation and utilization of VAB features, as (1) appearances may vary significantly, for example, between left and right profile views 21 ; and (2) VAB features are related to specific profiles, such as facial 2 ; 8 or side profile 4 . This motivates a reevaluation of current 2D detection-based monitoring approaches, as for a more complete phenotypical profiling, it is crucial to associate visual features with the corresponding side of the animal. One solution is monocular 3D object detection, i.e., lifting 2D bounding boxes to 3D cuboids in image space (Figure 1 ). This process performs not only 3D object detection but also monocular 6-degree-of-freedom (DoF) pose estimation for cuboid orientation and localization. These are challenging tasks in 3D scene understanding, with active research in autonomous driving, robotic manipulation, and AR 3 ; 5 . Analogous to autonomous driving, wildlife monitoring using Unmanned Aerial Vehicles (UAV) is steadily evolving toward autonomous functionality 10 ; 16 . In this context, image-based 3D animal detection could enable UAVs to autonomously reposition in real-time to capture optimal animal profile viewpoints, while also supporting downstream data analysis. Applying 3D detection methods to animals, however, adds further complexity due to non-rigid shapes, diverse poses and appearances, and cluttered environments. Figure 1 : Illustrative description of lifting 2D detection to oriented 3D-bounding boxes. 3D annotation remains a major bottleneck; manually labeling 3D boxes in images is nearly impossible without auxiliary 3D data 18 ; 17 . Current tools often rely on RGB-D sequences or LiDAR-enhanced point clouds 24 ; 9 . Even newer purely RGB-based approaches tend to support only rigid objects of known shapes or almost always require manual perspective snapping, known camera intrinsics, or video sequences, impractical for in-the-wild animal images with arbitrary viewpoints 23 ; 20 . Introducing new object classes also suffers from a cold-start problem: without high-quality 3D bounding box annotations, it is difficult to train or validate models to assist labeling for monocular 3D animal detection (Figure 2 ). Thus, leveraging existing curated RGB-only datasets to produce viewpoint consistent oriented 3D boxes remains highly challenging. Figure 2 : Zero shot predictions from Ovmono3D 26 on Imagenet samples. A promising avenue is to utilize Skinned Multi-Animal Linear (SMAL) 27 models as auxiliary 3D evidence, especially with the recent release of datasets like Animal3D 25 . However, directly projecting mesh-derived 3D boxes onto images often results in misaligned or incorrect orientations due to: (1) small manual annotation errors propagating into large pose estimation errors, (2) insufficient global constraints from 3D-to-2D correspondences, often leading to degenerate or depth-flipped boxes, (3) unreliable orientation estimates from ubiquitous methods like Principal Component Analysis (PCA) in the presence of complex poses and occlusion. We propose a pipeline for generating high-quality 3D bounding box annotations while overcoming the above limitations and leveraging SMAL to override the need for multi-sensor data. The generated labels aim to support the development of monocular 3D detection pipelines and deep learning-assisted labeling workflows. To provide view-specific information for VAB applications, we augment the labels by introducing an animal profile visibility metric. We evaluate our 3D bounding box label-generation pipeline on the Animal3D dataset and UAV-captured data, demonstrating robust label generation across diverse viewpoints. 2 Methodology We present an end-to-end pipeline that leverages SMAL models to generate 3D bounding box labels on images. An overview of the methodology is shown in Figure 3 . Given an input image and corresponding SMAL model, the algorithm extracts an orientation-aware 3D bounding box and a refined camera pose. These are combined to form a 3D box representation in the 2D image along with the profile visibility metric. Due to the reliance on SMAL models as auxiliary 3D reference data for corresponding images, an inherent limitation is that the 3D bounding box labels can only be generated for the animal species that are representable within the SMAL feature space. Figure 3 : Pipeline overview. Regarding inputs, it is important to clarify that the SMAL model fitting inherently requires 2D keypoint annotations as part of its standard pipeline. These keypoints establish critical correspondence between the input image and 3D model, enabling the optimization process to infer 3D keypoints during fitting. The 3D keypoints emerge naturally as a byproduct of the SMAL fitting process, rather than requiring separate generation as additional system inputs. 2.1 Orientation-aware 3D bounding box generation 2.1.1 Orientation To generate correctly oriented 3D bounding boxes from SMAL, we use a targeted method for defining consistent bounding box axes across meshes. This was motivated by the failure of PCA to generalize across species and poses, due to: (i) skew from non-representative features like tails, as PCA does not prioritize semantically meaningful regions; (ii) axis ambiguity from symmetric structures, causing eigenvector flips; and (iii) misalignment between mesh vertex distributions and the natural orientation of animals in complex poses. To address this, we use 3D landmark keypoints and their image-space visibility to create a fallback mechanism for selecting stable axes. The x-axis is defined along the anterior-posterior direction, the y-axis along the left-right direction, and the z-axis is derived using Fleming’s left-hand rule. 2.1.2 3D Bounding Box Generation The bounding box dimensions and position are calculated using a standard coordinate system. Mesh vertices are converted to local coordinate system of the anatomically aligned rotation matrix R_{\text{box}}^{T} , using V_{\text{local}}=R_{\text{box}}^{T}(V_{\text{world}}-c) where c=\frac{1}{N}\sum_{i=1}^{N}V_{i} is the centroid. The minimum and maximum coordinates of the box are calculated from the transformed mesh vertices using (V_{\text{local}}^{\min},\,V_{\text{local}}^{\max})=\Bigl(\min(V_{\text{local}})-\epsilon,\;\max(V_{\text{local}})+\epsilon\Bigr) with an optional small margin \epsilon . We maintain this small margin

Read the complete text