Interaction-via-Actions: Cattle Interaction Detection with Joint Learning of Action-Interaction Latent Space

Ren Nakagawa, Yang Yang, Risa Shinoda, Hiroaki Santo, Kenji Oyama, Fumio Okura, Takenao Ohkawa · 2025

Read Interaction-via-Actions: Cattle Interaction Detection with Joint Learning of Action-Interaction Latent Space by Ren Nakagawa, Yang Yang, Risa Shinoda, Hiroaki Santo, Kenji Oyama, Fumio Okura, Takenao Ohkawa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16133.txt.

Introduction

Ren Nakagawa and six co-authors teach computers to spot cattle behavior in work posted to arXiv in 2025, accepted to WACV 2026. Knowing when grazing cattle interact matters for farm management, for example to detect when a cow is in heat. Interactions are rare, though, so there is little labeled data showing two animals acting together.

Their method, CattleAct, breaks each interaction into the separate actions of the two animals. It first learns an action space from a large data set of single-animal behavior, then fine-tunes that space with a few interaction examples so actions and interactions share one map. A working system combines video and GPS inputs. On a commercial-scale pasture, CattleAct detected interactions more accurately than baseline methods, and the code is publicly available.

Catalog record

CREATORRen Nakagawa, Yang Yang, Risa Shinoda, Hiroaki Santo, Kenji Oyama, Fumio Okura, Takenao Ohkawa
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-16133

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interaction-via-Actions: Cattle Interaction Detection with Joint Learning of Action-Interaction Latent Space Ren Nakagawa, Yang Yang, Risa Shinoda, Hiroaki Santo, Kenji Oyama, Fumio Okura, Takenao Ohkawa Abstract This paper introduces a method and application for automatically detecting behavioral interactions between grazing cattle from a single image, which is essential for smart livestock management in the cattle industry, such as for detecting estrus. Although interaction detection for humans has been actively studied, a non-trivial challenge lies in cattle interaction detection, specifically the lack of a comprehensive behavioral dataset that includes interactions, as the interactions of grazing cattle are rare events. We, therefore, propose CattleAct, a data-efficient method for interaction detection by decomposing interactions into the combinations of actions by individual cattle. Specifically, we first learn an action latent space from a large-scale cattle action dataset. Then, we embed rare interactions via the fine-tuning of the pre-trained latent space using contrastive learning, thereby constructing a unified latent space of actions and interactions. On top of the proposed method, we develop a practical working system integrating video and GPS inputs. Experiments on a commercial-scale pasture demonstrate the accurate interaction detection achieved by our method compared to the baselines. Our implementation is available at https://github.com/rakawanegan/CattleAct . † † † Authors contributed equally. 1 Introduction Figure 1 : Joint estimation of actions and interactions of cows. Given a single-image observation, we estimate both frequent actions ( e.g . , standing, shown in blue) by individual cows and rare interactions ( e.g . , mount, shown in purple) by two cows, leveraging the joint latent space of actions and interactions. Monitoring cattle behavior is a critical component of modern livestock management, encompassing the detection of a wide range of events such as feeding, resting, social interactions, and reproductive activities, which consist of actions by single cows and interactions between two cows. Traditionally, the monitoring task has relied on farmers’ visual observation, which is labor-intensive and difficult to scale in commercial farms. Automating behavioral monitoring through computer vision and sensor technologies has thus become an important research challenge. Image or video-based action recognition has been extensively studied in the human domain [ 29 , 32 ] , where abundant high-quality video datasets [ 11 , 12 ] and well-annotated benchmarks [ 5 ] have driven the development of increasingly accurate and efficient models. These methods typically rely on spatio-temporal reasoning over dense action sequences to capture motion cues. Extending to animal behavior analysis, many studies have employed pipelines similar to human action recognition for wildlife monitoring [ 1 , 2 , 8 ] , livestock management [ 16 , 41 , 40 , 28 ] , and behavioral research [ 20 , 21 ] . Most methods process short video clips using models such as 3DCNNs [ 39 , 22 ] or transformer-based video encoders [ 1 , 8 ] to capture visual features while incorporating pose cues [ 41 , 23 ] . However, in commercial farm practice, interaction events of multiple cows, such as mounting or conflicts, occur rarely compared to individual actions like grazing or lying. This long-tail distribution is inevitable in real herds and cannot be solved simply by scaling up data collection, since even continuous recording over extended periods yields only a few positive samples of interactions. Therefore, interaction detection that effectively leverages abundant individual action data for cows is crucial for automated livestock production. To address these, we propose CattleAct, a data-efficient framework that decomposes interactions into the constituent actions of individual cattle, resulting in the joint estimation of actions and interactions of cows as shown in Fig. 1 . We first construct an action latent space by pre-training on a large collection of individual action data. We then embed rare interaction cases by fine-tuning the pre-trained latent space with contrastive learning, aligning action and interaction latents within a unified space. This alignment is expected to provide mutual regularization; i.e . , action features constrain interaction predictions, while interaction cues provide context to resolve ambiguities in individual actions. While video provides temporal information, constructing large-scale video datasets with precise temporal annotations is costly. Additionally, continuously storing and processing video streams is unrealistic in real-world farm environments due to storage, computational, and communication constraints. Since the target behaviors are recognizable from single frames, we design our method for image-based inputs, which enables scalable annotation, efficient training, and practical deployment. This approach supports continuous monitoring in a cost-effective manner by leveraging lightweight inference from sampled frames. On top of the proposed method, we develop a practical system for a production-scale pasture, applying a data augmentation method that preserves semantic information by referencing skeletal information. Experiments on a pasture of Japanese Black cattle demonstrate that aligning action and interaction embeddings significantly improves the recognition accuracy compared to baselines that model them independently or those trained on interactions alone. Contributions Our principal contributions are threefold: • We introduce a novel framework that decomposes rare interactions into combinations of individual actions and aligns their embeddings in a unified feature space, enabling data-efficient training. • We propose a data augmentation strategy that artificially simulates occlusion between overlapping cattle, improving the robustness in dense pasture environments. • We develop and deploy a practical multimodal system that integrates video and GPS data for automatic cattle interaction detection, and demonstrate its effectiveness in a commercial pasture setting. Figure 2 : Overview of CattleAct . (a) System overview: detected and tracked cows are re-identified, split into individuals and interaction candidates, and encoded by the action and interaction encoders, whose features are aligned by a multi-head attention module for final action–interaction classification. (b) GPS–image matching-based re-identification aligns GPS trajectories with detection trajectories to maintain consistent cattle IDs. (c) Contrastive-based action–interaction feature alignment pulls features of interacting pairs together while pushing apart negative and no interaction pairs in the joint latent space. 2 Related Work In-the-wild animal monitoring Automated monitoring of wild animals [ 1 , 8 ] is an active research field in ecology and biodiversity conservation. Large-scale datasets such as MammalNet [ 6 ] and Snapshot Serengeti [ 31 ] have enabled the development of species classification and population estimation models at unprecedented scales. These datasets typically contain millions of images spanning diverse habitats and species, providing strong benchmarks for visual recognition under uncontrolled conditions. Unlike livestock monitoring, their primary objective is species-level identification and distribution analysis, rather than fine-grained recognition of individuals or social interactions. The imaging setups in wildlife projects are typically sparse and rely on static camera traps over broad habitats, whereas farm setups place fixed cameras to capture dense herds at closer range and larger scale. Livestock action detection In contrast, livestock monitoring focuses on individual-level analysis to improve farm management and animal welfare. Traditional approaches rely on wearable or implantable sensors [ 3 ] such as GPS collars, RFID tags, or accelerometers, which are reliable for coarse activity recognition ( e.g . , feeding, locomotion) and long-term behavioral tracking. Nevertheless, such methods are limited in their ability to capture subtle actions or complex social interactions, while also incurring costs for deployment, maintenance, and data synchronization. Vision-based systems emerge for low-cost, scalable observation of livestock behaviors. With advances in computer vision, techniques for animal detection [ 15 ,

Read the complete text