From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification
Vasiliy Kudryavtsev , Kirill Borodin *, German Berezin, Kirill Bubenchikov, Grach Mkrtchian *and Alexander Ryzhkov · 2026
Read From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification by Vasiliy Kudryavtsev , Kirill Borodin *, German Berezin, Kirill Bubenchikov, Grach Mkrtchian *and Alexander Ryzhkov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02270.txt.
Introduction
Vasiliy Kudryavtsev, Kirill Borodin, German Berezin, and colleagues build a pet identification system in this 2026 study. Reuniting lost pets with owners needs fast, accurate matching from photos. Photo-only systems struggle, and past data sets were small. The team adds fake text descriptions as extra identity clues.
Their training corpus holds almost two million photos of near seven hundred thousand animals. Careful tests picked the best image and text backbones for the job. A gated fusion step, which weighs each source before merging, worked best. The full system reached top accuracy well above photo-only baselines. Fake text descriptions sharpened the line between similar animals, the results show. The work points to richer profiles for lost-pet searches.
Catalog record
| CREATOR | Vasiliy Kudryavtsev , Kirill Borodin *, German Berezin, Kirill Bubenchikov, Grach Mkrtchian *and Alexander Ryzhkov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97589 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.02270
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification Vasiliy Kudryavtsev , Kirill Borodin *, German Berezin, Kirill Bubenchikov, Grach Mkrtchian *and Alexander Ryzhkov Abstract Automated animal identification is a practical task for reuniting lost pets with their owners, yet current systems often struggle due to limited dataset scale and reliance on unimodal visual cues. This study introduces a multimodal verification framework that enhances visual features with semantic identity priors derived from synthetic textual descriptions. We constructed a massive training corpus of 1.9 million photographs covering 695,091 unique animals to support this investigation. Through systematic ablation studies, we identified SigLIP2-Giant and E5-Small-v2 as the optimal vision and text backbones. We further evaluated fusion strategies ranging from simple concatenation to adaptive gating to determine the best method for integrating these modalities. Our proposed approach utilizes a gated fusion mechanism and achieved a Top-1 accuracy of 84.28% and an Equal Error Rate of 0.0422 on a comprehensive test protocol. These results represent an 11% improvement over leading unimodal baselines and demonstrate that integrating synthesized semantic descriptions significantly refines decision boundaries in large-scale pet re-identification. keywords animal biometrics; pet reunification; multimodal deep learning; visual-semantic fusion; SigLIP; vision–language models; synthetic text generation; metric learning † † firstpage: 1 † † volume: 12 † † issue: 1 † † articlenumber: 30 † † year: 2026 † † copyright-year: 2026 † † external-editor: Constantine Kotropoulos † † received: 4 December 2025 † † revised: 22 December 2025 † † accepted: 4 January 2026 † † published: 7 January 2026 † † authornames: Vasiliy Kudryavtsev, Kirill Borodin, German Berezin, Kirill Bubenchikov, Grach Mkrtchian and Alexander Ryzhkov † † address: 1 Faculty of IT, Technical University of Communication and Informatics, Moscow, Russia, 111024; [email protected] (V.K.); [email protected] (G.B.) 2 AI lab, Avito, Moscow, Russia, 125196; [email protected] (K.B.); [email protected] (A.R.) † † corresponding: Correspondence: [email protected] (K.B.); [email protected] (G.M.) † † reftitle: References 1 Introduction 1.1 Context and Relevance Individual animal identification is a notable problem across multiple domains that demand reliable and automated solutions ( Zhang et al. 2024 ; Vidal et al. 2021 ) . Reunification of lost pets ( Weiss et al. 2012 ) , wildlife conservation tracking ( Vidal et al. 2021 ; Zheng et al. 2022 ) , livestock management ( Zheng et al. 2022 ; Kang and Oh 2025 ) , and veterinary care ( Vidal et al. 2021 ; Zhang et al. 2024 ) all depend on accurate identification systems. Traditional approaches have developed physical markers such as microchips and collar tags ( Lord et al. 2010 ; McGreevy et al. 2019 ) to address this need, yet these methods introduce meaningful practical limitations. Physical markers can fall out, break, or become lost during the animal’s lifetime ( Lord et al. 2010 ; Lancaster et al. 2015 ) , rendering them unstable in real-world scenarios. These constraints have motivated exploration of alternative identification methods that do not depend on auxiliary physical devices ( Wahltinez and Wahltinez 2024 ; Vidal et al. 2021 ; Zheng et al. 2022 ) . Visual recognition systems offer a practical alternative that circumvents the limitations of physical markers by leveraging the unique biometric features inherent to each animal ( Matley et al. 2024 ; Petso et al. 2022 ) . Rather than relying on attached or implanted devices, visual-based systems capture facial and body characteristics that remain stable throughout an animal’s life ( Morandi et al. 2022 ; Meenakshisundaram et al. 2021 ) . This approach proves particularly valuable in field scenarios where physical markers may be inaccessible or have been removed ( Matley et al. 2024 ; McCutcheon et al. 2025 ) . Deep learning techniques have enabled scalable processing of visual information by allowing systems to extract and compare biometric features from camera images without human intervention ( Cheema et al. 2021 ; Schofield et al. 2019 ) . Deep learning-based visual identification systems provide noticeable advantages for animal identification tasks ( Zhang et al. 2024 ; Vidal et al. 2021 ) . These approaches are non-invasive, requiring only image capture rather than physical contact or device implantation ( Wahltinez and Wahltinez 2024 ; Mougeot et al. 2019 ; Ferreira et al. 2020 ) . The methods are inherently scalable, as they can process multiple subjects simultaneously and adapt to new individuals without requiring system reconfiguration ( Romero-Ferrero et al. 2019 ; Sakamoto et al. 2023 ; Cheema et al. 2021 ) . In contrast to breed-level classification systems, which categorize animals into predefined species or breed categories, individual identification systems must distinguish between unique subjects within a population ( Vidal et al. 2021 ; Zhang et al. 2024 ; Hou et al. 2025 ) . This more granular level of recognition enables finer-grained management and tracking across diverse real-world deployment scenarios ( de Lorm et al. 2023a ; Zheng et al. 2022 ) . However, the development of robust visual identification systems faces considerable technical challenges ( Hou et al. 2025 ; de Lorm et al. 2023a ) . Systems must contend with limited training datasets, where the number of individual animals with sufficient labeled image samples remains constrained ( de Lorm et al. 2023a ; Nepovinnykh et al. 2022 ) . Annotation quality and consistency present additional difficulties, particularly when labels are collected across multiple sources or by different annotators ( Nepovinnykh et al. 2022 ; Cermak et al. 2024 ) . A primary limitation emerges from model generalization failures on new animals not encountered during training ( Hou et al. 2025 ; Adam et al. 2025 ) . Models trained on specific populations may fail to identify individuals from different environments, with varying lighting conditions, camera angles, or image quality ( Odo et al. 2025 ; Beery et al. 2018 ) . These generalization failures compromise system reliability in production environments where the model must operate across diverse conditions without retraining ( Hou et al. 2025 ; Santamaria et al. 2025 ) . Research addressing individual animal identification therefore requires systematic investigation of how to build systems that maintain robust performance across these diverse real-world conditions ( Zhang et al. 2024 ; Hou et al. 2025 ) . This investigation must encompass dataset construction strategies that capture natural variation ( Cermak et al. 2024 ; Otarashvili et al. 2024 ) , architectural choices that generalize effectively ( Hou et al. 2025 ; Schneider et al. 2020 ; Wahltinez and Wahltinez 2024 ) , and evaluation protocols that measure performance realistically ( Hou et al. 2025 ; de Lorm et al. 2023a ) . Understanding which approaches enable reliable identification across different populations, environments, and deployment scenarios ( de Lorm et al. 2023a ) remains essential for enabling widespread adoption of visual identification systems in practical applications ( Hou et al. 2025 ; Zhang et al. 2024 ) . 1.2 Research Problem and Associated Challenges Individual animal identification differs fundamentally from breed-level classification and species recognition tasks ( Zhang et al. 2024 ) . While classification systems assign animals to predefined categories based on taxonomic or morphological characteristics ( Ratnasingham and Hebert 2013 ) , individual identification requires distinguishing between unique subjects within a population ( Vidal et al. 2021