BehaviorVLM: Unified Finetuning-Free Behavioral Understanding with Vision-Language Reasoning
Jingyang Ke, Weihan Li, Amartya Pradhan, Jeffrey E. Markowitz, Anqi Wu · 2026
Read BehaviorVLM: Unified Finetuning-Free Behavioral Understanding with Vision-Language Reasoning by Jingyang Ke, Weihan Li, Amartya Pradhan, Jeffrey E. Markowitz, Anqi Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12176.txt.
Introduction
Ke, Li, Pradhan, Markowitz, and Wu read animal behavior with vision-language models and no task finetuning. Pose tracking and behavior reading both lean on heavy human labels in current practice. BehaviorVLM guides pretrained models through explicit steps that a user can verify. For pose, multi-stage reasoning across time, space, and views flags weak labels through geometric checks. For behavior, clustering finds candidate segments, a vision model captions each clip, and a language model merges and names them. The pipeline scales to multi-animal data with little labeling.
Catalog record
| CREATOR | Jingyang Ke, Weihan Li, Amartya Pradhan, Jeffrey E. Markowitz, Anqi Wu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28631 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12176
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BehaviorVLM: Unified Finetuning-Free Behavioral Understanding with Vision-Language Reasoning Jingyang Ke, Weihan Li, Amartya Pradhan, Jeffrey E. Markowitz, Anqi Wu Abstract Understanding freely moving animal behavior is central to neuroscience, where pose estimation and behavioral understanding form the foundation for linking neural activity to natural actions. Yet both tasks still depend heavily on human annotation or unstable unsupervised pipelines, limiting scalability and reproducibility. We present BehaviorVLM , a unified vision-language framework for pose estimation and behavioral understanding that requires no task-specific finetuning and minimal human labeling by guiding pretrained Vision-Language Models (VLMs) through detailed, explicit, and verifiable reasoning steps. For pose estimation, we leverage quantum-dot-grounded behavioral data and propose a multi-stage pipeline that integrates temporal, spatial, and cross-view reasoning. This design greatly reduces human annotation effort, exposes low-confidence labels through geometric checks such as reprojection error, and produces labels that can later be filtered, corrected, or used to fine-tune downstream pose models. For behavioral understanding, we propose a pipeline that integrates deep embedded clustering for over-segmented behavior discovery, VLM-based per-clip video captioning, and LLM-based reasoning to merge and semantically label behavioral segments. The behavioral pipeline can operate directly from visual information and does not require keypoints to segment behavior. Together, these components enable scalable, interpretable, and label-light analysis of multi-animal behavior. 1 Introduction Understanding freely moving animal behavior is central to neuroscience. Two fundamental tasks are pose estimation and behavioral segmentation, which together provide the bridge between neural activity and natural action. In practice, however, both problems still require substantial human labor. Pose estimation toolkits such as DeepLabCut ( 6 ) , SLEAP ( 7 ) , and Lightning Pose ( 2 ) can achieve strong accuracy, but each new experimental setup usually requires manual labels before training can begin. Pretrained foundation models such as SuperAnimal ( 16 ) reduce this burden, yet they still depend on human-labeled pretraining data and can degrade under new camera geometries, imaging conditions, or animal morphologies. Behavioral understanding faces a parallel limitation. In this paper, behavioral understanding refers to behavioral segmentation together with a human-understandable interpretation for each segment. Recent VLM- and LLM-based systems such as MouseGPT ( 15 ) and AmadeusGPT ( 17 ) show that language models can help describe animal behavior, but they do not replace the full annotation workflow that a human analyst performs when identifying transitions and assigning semantic labels to behavior segments. At the other extreme, unsupervised approaches such as MotionMapper ( 1 ) , MoSeq ( 13 ) , and Keypoint-MoSeq ( 12 ) scale well, but they often produce segments that are difficult to interpret, switch too rapidly, or do not align cleanly with human-understandable behavioral categories. This limitation arises because these methods typically rely on keypoints or low-dimensional motion representations and do not directly extract semantic labels from the visual evidence in the video. Figure 1: Overview of BehaviorVLM. This VLM & LLM-based framework addresses pose estimation and behavioral understanding with minimum manual labeling and no finetuning. We present BehaviorVLM , a unified vision-language framework that addresses both pose estimation and behavioral understanding without task-specific finetuning and with minimal human labeling , by guiding pretrained Vision-Language Models (VLMs) through structured, multi-stage reasoning pipelines. The central idea is to mimic how a human would carry out these annotation tasks in practice. Rather than asking a model for a final answer in a single step, we decompose each task into explicit intermediate stages that use visual evidence, expose uncertainty, and allow labels to be reviewed or corrected afterward. This framing is especially useful when the goal is to replace large amounts of manual work rather than to claim that every automatic label is perfect. For pose estimation , we leverage near-infrared fluorescent quantum dots (QDs) ( 11 ) injected at body keypoints to provide candidate keypoint locations across six synchronized camera views. A VLM is guided through a multi-stage reasoning pipeline that integrates temporal, spatial, and cross-view constraints to predict accurate 3D keypoint trajectories. The pipeline requires only three manually labeled seed frames, and completed predictions are appended to a rolling window and reused as few-shot exemplars for later frames. The QD signals substantially reduce human labeling effort compared with conventional manual annotation, reduce bias from imprecise human labels, and make it possible to identify poor pseudo-labels after the fact using geometric criteria such as large 3D reprojection error. Those filtered labels can then be used directly or used to fine-tune a downstream pose estimation model. More broadly, this setup encourages the use of QD-based labeling for small animals such as mice, fish, and birds, where conventional pose annotation or motion tracking with motion capture devices is especially difficult. For behavioral understanding , we introduce a multi-stage pipeline that first applies deep embedded clustering to obtain fine-grained, over-segmented behavioral clips for each animal, then invokes a VLM to generate per-clip behavioral labels and natural-language descriptions, and finally leverages an LLM to merge similar segments and assign semantically meaningful labels. This pipeline makes heavy use of visual information. In particular, the segmentation process can operate directly on video features and does not require keypoints, which distinguishes it from prior behavior pipelines that are restricted to pose-based inputs. Together, these two pipelines form a unified framework (Figure 1 ) that replaces extensive human annotation and task-specific model training with structured vision-language reasoning, enabling scalable and interpretable automated analysis of naturalistic animal behavior. Our main contributions are: • A multi-stage VLM-based reasoning pipeline for QD-grounded pose estimation that requires only three labeled seed frames and produces labels that can be inspected, filtered, corrected, and reused for downstream pose model fine-tuning. • A multi-stage behavioral understanding pipeline that converts visual or fused behavioral features into semantically meaningful behavioral segments through low-cost over-segmentation, VLM-based visual interpretation, and LLM-based semantic reasoning. • Evaluation on a custom six-view quantum-dot mouse dataset ( 11 ) and the MABe2022 Mouse Triplets benchmark ( 10 ) , demonstrating that finetuning-free vision-language reasoning can achieve reliable pose estimation and interpretable multi-animal behavioral segmentation. 2 Pose Estimation 2.1 Experimental Data We use a dataset of 500 synchronized timepoints from six cameras recording a freely moving mouse. The mouse was injected with near-infrared fluorescent nanoparticles (quantum dots, QDs) at 12 anatomical keypoints, following the QD data acquisition procedure in ( 11 ) . This setup provides both reflectance images, which capture the visible behavior of the animal, and fluorescence images, which reveal the QD signals at body locations (Figure 2 A). Each fluorescence centroid indicates the location of a body marker, but not its anatomical identity. The raw frames have resolution 2048\times 1400 in every view. For each frame and camera, we apply Segment Anything 3 ( 3 ) to detect the mouse body mask. We then crop the frame to the mask’s tight bounding box and pad the crop by 16 pixels plus 5% of the bounding box dimensions. Because the apparent size and position of the mouse vary across viewpoints and over time, the crop dimensions differ across cameras and frames. QD centroid locations are extracted from the fluorescence channel inside each crop and overlaid as numbered candidate keypoints on the reflectance image (Figure 2 B). The task is to assign anatomical identities to these candidate points with minimal