Transforming Faces Into Video Stories-VideoFace2.0
Branko Brkljač 1 4 1 , Vladimir Kalušev 2 4 , Branislav Popović 1 , Milan Sečujski 1 · 2025
Read Transforming Faces Into Video Stories-VideoFace2.0 by Branko Brkljač 1 4 1 , Vladimir Kalušev 2 4 , Branislav Popović 1 , Milan Sečujski 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02060.txt.
Introduction
Branko Brkljač and co-authors present a 2025 paper on VideoFace2.0, a tool that turns raw video into structured stories organized by the faces in it. The name honours an early digitizer that first let users capture video freely. The modern system finds every distinct face, follows it across time and shots, and catalogues who appears where.
Built to run nearly in real time, it also characterizes the faces it tracks and emits tidy outputs for later analysis. The authors aim it at television production and media analysis, and at building large video datasets. Face re-identification, once a research curiosity, becomes an indexing workhorse for archives.
Catalog record
| CREATOR | Branko Brkljač 1 4 1 , Vladimir Kalušev 2 4 , Branislav Popović 1 , Milan Sečujski 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02060
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transforming Faces Into Video Stories-VideoFace2.0 Thanks: 4 equal contribution Branko Brkljač 1 4 1 , Vladimir Kalušev 2 4 , Branislav Popović 1 , Milan Sečujski 1 Abstract Face detection and face recognition have been in the focus of vision community since the very beginnings. Inspired by the success of the original Videoface digitizer, a pioneering device that allowed users to capture video signals from any source, we have designed an advanced video analytics tool to efficiently create structured video stories, i.e. identity-based information catalogs. VideoFace2.0 is the name of the developed system for spatial and temporal localization of each unique face in the input video, i.e. face re-identification (ReID), which also allows their cataloging, characterization and creation of structured video outputs for later downstream tasks. Developed near real-time solution is primarily designed to be utilized in application scenarios involving TV production, media analysis, and as an efficient tool for creating large video datasets necessary for training machine learning (ML) models in challenging vision tasks such as lip reading and multimodal speech recognition. Conducted experiments confirm applicability of the proposed face ReID algorithm that is combining the concepts of face detection, face recognition and passive tracking-by-detection in order to achieve robust and efficient face ReID. The system is envisioned as a compact and modular extensions of the existing video production equipment. Presented results are based on test implementation that achieves between 18–25 fps on consumer type notebook. Ablation experiments also confirmed that the proposed algorithm brings relative gain in the reduction of number of false identities in the range of 73%–93%. I Introduction Video content has become the prevailing type of global internet traffic [ 1 ] . Similarly, media productions that include news, podcasts, and interviews had been steadily growing in the new internet formats. Due to the large amount of available data, different types of advanced machine learning (ML) vision tasks involving human speakers had also come into the focus of research community. However, availability of adequate video analysis tools represents a technological burden for more effective mastering of the respective tasks. Inspired by the historical success of the original Videoface digitizer [ 2 ] we have developed VideoFace2.0, Fig. 1 . The main assumption is that analyzed video content, regardless of source and application, contains appearance of multiple persons or speakers that randomly enter the scene during the video, with face orientation towards the camera. The main goal of the system is to efficiently perform spatial and temporal localization of each unique face in the input video, i.e. solve the face re-identification (ReID) task and provide functionalities for cataloging, characterization and production of structured video outputs or video stories, Fig. 1 . It is assumed that neither the identities nor the number of people are known in advance. Fig. 1 : VideoFace 2.0 processing workflow and applications. Input video from recorded file, network stream or camera feed is uploaded into the system and analyzed frame-by frame in order to detect human faces (spatial and temporal localization), catalog their identity (create a new entry, i.e. video story or re-identify the person if it was already present in the video before). It can also generate additional personal characteristics with specific time stamps (e.g. facial landmarks, facial dynamics, age, gender and audio/speech). The system is designed to operate in near real-time mode and allows for standalone embedded hardware implementation. VideoFace2.0 is primarily designed to be utilized in application scenarios that include TV production, media analysis and creative industries, where video stories generated by the system would be an efficient way for fast and easier video montage of scenes in which specific persons of interest appear. E.g. editing of interviews or reportages brought by the field reporters to TV studios, talk-shows, podcasts and other formats that include multiple speakers or participants. Another important application field includes production of custom video-based datasets that are required in different ML tasks involving multi-modal inputs like speech, text and image. Challenging vision tasks like lip reading and multi-modal speech recognition require large-scale language and speaker specific video datasets necessary for ML model training. Thus, one of the main goals of VideoFace2.0 and the proposed functionality and designed elements was also to facilitate creation of high-quality multi-modal datasets involving human speakers. The main contribution of the paper is the generic and modular algorithm for face ReID, which is combining the concepts of face detection, recognition and passive tracking-by-detection. The rest of the paper is organized as follows. In Section II we introduce the main vision tasks behind the proposed processing workflow and discuss current state-of-the-art in the field. Next, in Section III we present the main technical and algorithmic elements of the proposed solution, including proposed person re-identification improvements. Section IV brings results of conducted experiments, performance evaluation and discussion of possible limitations. Finally, in Section V we reexamine the strategies for further improvements and provide a reference for future work and planned applications. Code implementation is available at: https://github.com/brkljac/VideoFace2.0 , while the results of test experiments are also available on the following link: https://youtube.com/@kalusev . II Related work The proposed processing workflow, Fig. 1 , is based on four vision tasks that have been extensively studied and analyzed in the literature. These are face detection [ 3 , 4 ] , face recognition [ 5 , 6 ] , face re-identification (ReID) [ 7 ] and object tracking. Since each of these tasks is characterized by specific performance metrics and comprehensive evaluation benchmarks that go beyond the scope of this presentation, in this paper we have decided to rely on off-the-shelf solutions that have been properly tested and proven as competitive to current state-of-the-art approaches in practice. In terms of applied algorithms and chosen pre-trained models we have followed the usual paradigm in which face detection results are used as a prerequisite for face characterization using learned face embeddings and followed by face identity matching in the corresponding high dimensional feature space during the face ReID phase. Nevertheless, we provide a brief overview of related works that illustrate design challenges and characteristics of the proposed VideoFace2.0. The main goal of face detection is to provide localization (bounding box) and probability (decision confidence score) of face appearance in the image. Since design requirements almost always are that face detection algorithm are fast and lightweight in terms of computational and memory resources, detection is usually performed over still images or individual video frames without the use of temporal information. It is a special case of object detection task, and as such depending on application requires the choice of optimal working point on the detector’s receiver operating characteristic [ 8 ] . The trade-off between face detection sensitivity and the level of false alarms (precision) is determined by detector’s role in processing workflow. Similarly to other object detectors [ 9 ] , traditionally face detection was also designed as a two stage process involving face localization (e.g. sliding window search or region proposal networks [ 10 ] ) and subsequent binary classification of region of interest (ROI). However, this changed with introduction of more efficient single stage detectors [ 11 , 12 ] . Open-set face ReID task is a challenging vision problem that is attracting significant attention