Person Recognition at Altitude and Range: Fusion of Face, Body Shape and Gait

Feng Liu, Nicholas Chimitt, Lanqing Guo, Jitesh Jain, Aditya Kane, Minchul Kim, Wes Robbins, Yiyang Su, Dingqiang Ye, Xingguang Zhang, Jie Zhu, Siddharth Satyakam, Christopher Perry, Stanley H. Chan, Arun Ross, Humphrey Shi, Zhangyang Wang, Anil Jain, Xiaoming Liu · 2025

Read Person Recognition at Altitude and Range: Fusion of Face, Body Shape and Gait by Feng Liu, Nicholas Chimitt, Lanqing Guo, Jitesh Jain, Aditya Kane, Minchul Kim, Wes Robbins, Yiyang Su, Dingqiang Ye, Xingguang Zhang, Jie Zhu, Siddharth Satyakam, Christopher Perry, Stanley H. Chan, Arun Ross, Humphrey Shi, Zhangyang Wang, Anil Jain, Xiaoming Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04616.txt.

Introduction

Feng Liu and a large team built FarSight in 2025, a system that names people at long range by fusing face, body shape and walking style. Surveillance cameras at distance and height face turbulence, wind and bad angles. The system chains four parts: finding and tracking people, restoring the video, encoding each cue, and merging scores by quality. Each part was tuned for degraded images.

On the hard BRIAR benchmark, the new system beat the team's earlier one by wide margins. Verification accuracy jumped by 34 points, and open-set mistakes fell by about a third. It also entered a national face-in-video evaluation run in 2025. The authors present FarSight as ready for real biometric work under harsh conditions. The gains come from treating the whole chain, not one cue, as the problem.

Catalog record

CREATORFeng Liu, Nicholas Chimitt, Lanqing Guo, Jitesh Jain, Aditya Kane, Minchul Kim, Wes Robbins, Yiyang Su, Dingqiang Ye, Xingguang Zhang, Jie Zhu, Siddharth Satyakam, Christopher Perry, Stanley H. Chan, Arun Ross, Humphrey Shi, Zhangyang Wang, Anil Jain, Xiaoming Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-04616

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Person Recognition at Altitude and Range: Fusion of Face, Body Shape and Gait Feng Liu, Nicholas Chimitt, Lanqing Guo, Jitesh Jain, Aditya Kane, Minchul Kim, Wes Robbins, Yiyang Su, Dingqiang Ye, Xingguang Zhang, Jie Zhu, Siddharth Satyakam, Christopher Perry, Stanley H. Chan, Arun Ross, Humphrey Shi, Zhangyang Wang, Anil Jain, Xiaoming Liu Abstract We address the problem of whole-body person recognition in unconstrained environments. This problem arises in surveillance scenarios such as those in the IARPA Biometric Recognition and Identification at Altitude and Range (BRIAR) program, where biometric data is captured at long standoff distances, elevated viewing angles, and under adverse atmospheric conditions ( e.g. , turbulence and high wind velocity). To this end, we propose FarSight , a unified end-to-end system for person recognition that integrates complementary biometric cues across face, gait, and body shape modalities. FarSight incorporates novel algorithms across four core modules: multi-subject detection and tracking, recognition-aware video restoration, modality-specific biometric feature encoding, and quality-guided multi-modal fusion. These components are designed to work cohesively under degraded image conditions, large pose and scale variations, and cross-domain gaps. Extensive experiments on the BRIAR dataset, one of the most comprehensive benchmarks for long-range, multi-modal biometric recognition, demonstrate the effectiveness of FarSight. Compared to our preliminary system [ 1 ] , this system achieves a 34.1% absolute gain in 1:1 verification accuracy ([email protected]% FAR), a 17.8% increase in closed-set identification (Rank-20), and a 34.3% reduction in open-set identification errors (FNIR@1% FPIR). Furthermore, FarSight was evaluated in the 2025 NIST RTE Face in Video Evaluation (FIVE), which conducts standardized face recognition testing on the BRIAR dataset. These results establish FarSight as a state-of-the-art solution for operational biometric recognition in challenging real-world conditions. I Introduction Unconstrained biometric recognition at long distances and elevated viewpoints is crucial for a variety of applications, including law enforcement, border security, wide-area surveillance, and public media analytics [ 2 , 3 , 4 ] . Among existing approaches, whole-body biometric recognition [ 1 , 5 , 6 , 7 , 8 , 9 ] has become a central focus in this domain, as it captures a rich combination of anatomical and behavioral traits—such as facial appearance, gait and body shape—offering greater resilience to occlusion, degradation, and modality loss than single-modality systems. Despite its potential, deploying whole-body recognition systems in real-world scenarios remains technically demanding. High-performing systems must not only incorporate robust multi-modal biometric modeling, but also support modules for precise person detection and tracking, enhancement of low-quality imagery, mitigation of atmospheric turbulence, and adaptive fusion strategies to handle unreliable data. Fig. 1: Illustration of the IARPA BRIAR whole-body image capture scenarios. (a) Enrollment Indoor Collection: High-quality still images and videos captured from multiple viewpoints under controlled conditions. (b) Probe Outdoor Collection: Videos captured in outdoor environments at varying distances and elevation angles, with challenging factors such as atmospheric turbulence. These settings reflect the real-world conditions encountered in long-range biometric recognition. Permission granted by the subject for use of imagery in publications . To develop and evaluate biometric systems that meet these demands, it is essential to have access to datasets that reflect the full complexity of real-world surveillance conditions. The IARPA Biometric Recognition and Identification at Altitude and Range (BRIAR) program 1 1 1 https://www.iarpa.gov/research-programs/briar is a collective effort [ 8 , 9 ] in this direction, fostering the development of biometric systems capable of performing reliably in these unconstrained scenarios. Fig. 1 illustrates the BRIAR whole-body image capture scenarios, comprising controlled indoor enrollment collections and challenging outdoor probe collections. These scenarios simulate the real-world challenges in person recognition, including: (i) low-quality video frames caused by long-range capture (up to 1000 meters) and atmospheric turbulence, with refractive index structure constant ranging from C_{n}^{2}=10^{-17} to 10^{-14}\,\text{m}^{-2/3} ; (ii) large yaw and pitch angles (up to 50^{\circ} ) from elevated platforms (drones) at altitudes up to 400 meters; (iii) degraded feature sets due to low visual quality, where the Inter-Pupillary Distance (IPD) ranges between 15–100 pixels; (iv) the complexity of open-set search, where probe images must be matched against galleries containing distractors; and (v) a significant domain gap caused by limited training data and the diversity of real-world conditions. To address the challenges posed by unconstrained, long-range biometric recognition, we propose FarSight , an integrated end-to-end system designed for robust person recognition using multi-modal biometric cues. FarSight combines face, gait, and body shape modalities to ensure recognition performance even when individual cues are unreliable or degraded. The system comprises four tightly coupled modules, each addressing a critical component of the recognition pipeline: (1) A multi-subject detection and tracking module that accurately localizes individuals in video sequences captured under dynamic, cluttered, and low-resolution conditions. (2) A recognition-aware video restoration module that mitigates visual degradation—particularly due to turbulence and long-range blur—by jointly optimizing image quality and biometric fidelity. (3) A biometric feature encoding module that extracts robust representations for each modality, leveraging recent advances in large vision models and modality-specific architectural designs. (4) A quality-guided multi-modal fusion module that adaptively integrates scores across modalities, accounting for variable input quality and partial observations. A preliminary version of our system [ 1 ] was previously presented at the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2024). Building on that foundation, we have substantially upgraded each module to improve recognition performance across verification, closed-set identification, and open-set search tasks. The current system also incorporates key architectural enhancements to support lower latency, reduced memory usage, and improved scalability. Below, we summarize the major improvements introduced in each module of the updated FarSight system: • Multi-Subject Detection and Tracking: Our initial system [ 1 ] employed a joint body-face detector based on R-CNN [ 10 , 11 ] , which lacked support for multi-subject tracking and exhibited high inference latency. To address these limitations, we introduce two key upgrades: First, we adopt a dual-detector framework using BPJDet [ 12 ] for coarse body-face localization followed by verification via YOLOv8 [ 13 ] to reduce false positives. This replacement improves both detection accuracy and runtime efficiency. Second, we develop PSR-ByteTrack, an enhanced multi-subject tracker built on ByteTrack [ 14 ] . PSR-ByteTrack mitigates issues such as ID switches, fragmented tracklets, and reidentification failures by introducing a patch-based retrieval mechanism that maintains subject-specific appearance features in memory. • Recognition-Aware Video Restoration : We introduce the Gated Recurrent Turbulence Mitigation (GRTM) network, a novel video-based restoration model tailored for long-range, turbulence-degraded imagery. A lightweight classifier is used to trigger restoration selectively, reducing unnecessary computation and avoiding potential feature distortion. A key contribution of this system is its tightly coupled restoration-recognition co-optimization framework which integrates recognition objectives directly into the restoration training process, guiding the model to enhance features critical for identity discrimination. • Biometric (Face, Gait and Body Shape) Feature Encoding: We upgrade each modality-specific model with task-aligned architectural improvements and training strategies tailored to the challenges of long-range, unconstrained biometric recognition. i) Face : We propose KP-RPE [ 15 ] , a keypoint-dependent relative position encoding technique which significantly improves the handling of misaligned and low-quality facial images. ii) Gait : We introduce BigGait [ 16 ] , the first gait recognition framework based on large vision models (LVMs). This method shifts from task-specific priors to general-purpose

Read the complete text