ORCA: O bject R ecognition and C omprehension for A rchiving Marine Species
Yuk-Kwan Wong, Haixin Liang, Zeyu Ma, Yiwei Chen, Ziqiang Zheng, Rinaldi Gotama, Pascal Sebastian, Lauren D. Sparks, Sai-Kit Yeung, [5pt] Hong Kong University of Science, Technology · 2025
Read ORCA: O bject R ecognition and C omprehension for A rchiving Marine Species by Yuk-Kwan Wong, Haixin Liang, Zeyu Ma, Yiwei Chen, Ziqiang Zheng, Rinaldi Gotama, Pascal Sebastian, Lauren D. Sparks, Sai-Kit Yeung, [5pt] Hong Kong University of Science, Technology (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21150.txt.
Introduction
Yuk-Kwan Wong, Sai-Kit Yeung, and seven co-authors give marine biology the benchmark it needs for automated surveys. ORCA gathers 14,647 images covering 478 marine species, annotated with more than forty-two thousand bounding boxes and over twenty-two thousand expert-verified captions that describe each animal's morphology in words. The dataset is built to align the field's real difficulties, look-alike species, cluttered scenes, scarce examples, with standard computer-vision tasks.
Eighteen state-of-the-art models are evaluated on closed-set and open-vocabulary detection, instance captioning, and visual grounding. Performance gaps expose exactly where general models fail underwater: species diversity, morphological overlap between relatives, and domain demands no land-based dataset teaches. Conservation programs that want scalable, automatic monitoring of ecosystems gain both training material and an honest measure of readiness.
Catalog record
| CREATOR | Yuk-Kwan Wong, Haixin Liang, Zeyu Ma, Yiwei Chen, Ziqiang Zheng, Rinaldi Gotama, Pascal Sebastian, Lauren D. Sparks, Sai-Kit Yeung, [5pt] Hong Kong University of Science, Technology |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-21150
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ORCA: O bject R ecognition and C omprehension for A rchiving Marine Species Yuk-Kwan Wong, Haixin Liang, Zeyu Ma, Yiwei Chen, Ziqiang Zheng, Rinaldi Gotama, Pascal Sebastian, Lauren D. Sparks, Sai-Kit Yeung, [5pt] Hong Kong University of Science, Technology Abstract Marine visual understanding is essential for monitoring and protecting marine ecosystems, enabling automatic and scalable biological surveys. However, progress is hindered by limited training data and the lack of a systematic task formulation that aligns domain-specific marine challenges with well-defined computer vision tasks, thereby limiting effective model application. To address this gap, we present ORCA, a multi-modal benchmark for marine research comprising 14,647 images from 478 species, with 42,217 bounding box annotations and 22,321 expert-verified instance captions. The dataset provides fine-grained visual and textual annotations that capture morphology-oriented attributes across diverse marine species. To catalyze methodological advances, we evaluate 18 state-of-the-art models on three tasks: object detection (closed-set and open-vocabulary), instance captioning, and visual grounding. Results highlight key challenges, including species diversity, morphological overlap, and specialized domain demands, underscoring the difficulty of marine understanding. ORCA thus establishes a comprehensive benchmark to advance research in marine domain. 1 Introduction The ocean, with a vast coverage on the surface of our blue planet, remains a mysterious abyssal region to the best of our knowledge. Advancing knowledge of marine ecosystems is critical for oceanography [ 11 , 80 ] , sustainable resource management [ 13 , 48 ] , and biodiversity conservation [ 43 , 37 ] . Considerable efforts have been devoted to biological surveys and habitat monitoring [ 78 , 71 , 2 , 67 ] . To enhance the scalability and efficiency of in-situ monitoring, researchers are increasingly leveraging computer vision techniques to reduce manual data processing, including image classification [ 79 , 62 , 8 ] , object detection [ 76 , 73 ] , and vision–language modeling [ 27 ] . Despite the remarkable success enabled by powerful network backbones and domain-specific datasets, significant challenges remain, which can be broadly categorized into issues of training data and task formulation . Current marine datasets are often restricted to a small set of predefined categories ( e.g. , seven semantic classes in the UIIS dataset [ 34 ] ) and are typically collected from limited geographic regions. Such constraints hinder both taxonomic diversity and ecological coverage, limiting the recognition of a broad range of marine species. Even fine-grained datasets with larger category sets, such as [ 88 , 89 ] , remain primarily focused on fish monitoring. Regarding task formulation, current task definitions remain insufficient for domain requirements. Image-level classification [ 79 , 62 , 8 ] may lead to inconsistency between coarse category annotations and image content, which often contains multiple species in a wild environment. Object detection [ 76 , 73 ] is limited by a narrow set of categories. Furthermore, category labels alone cannot capture key biological traits that are essential for ecological monitoring. For image captioning , although recent vision–language models (VLMs) [ 70 , 30 , 29 , 85 ] are evolving rapidly, their outputs are typically coarse and lack the granularity and domain-specific knowledge needed for description. To address these challenges, we introduce ORCA, the first multimodel dataset explicitly designed for marine research. ORCA offers 1) broad taxonomic coverage spanning 478 species and 670 common-name categories; 2) instance-level annotations enabling both object detection and grounding; and 3) biology-oriented captions with diagnostic traits, appearances, behaviors, and habitats, all validated by marine biologists. The dataset comprises 14,647 images with 42,217 bounding boxes, each labeled with both scientific and common names to support diverse usage scenarios. In total, ORCA provides 22,321 expertly verified instance–caption pairs, ensuring terminological accuracy and scientific relevance. ORCA supports a range of vision-language tasks, including closed-set and open-vocabulary detection, instance-level captioning, and visual grounding. While detection and grounding primarily assess a model’s ability to recognize and localize marine species, ORCA further introduces three evaluation settings: Class-Level , Intra-Class , and Inter-Class , to systematically examine how taxonomic hierarchies influence those abilities under the condition of morphological overlapping , where closely related species exhibit highly similar traits, thereby complicating species identification. Beyond spatial localization, the captioning and grounding components of ORCA facilitate fine-grained alignment between visual observations and linguistic descriptions. This dual emphasis not only enhances object-level referencing but also supports the structured, biologically meaningful archiving of marine survey data. We have benchmarked 18 state-of-the-art algorithms across the aforementioned tasks. In summary, our contributions can be outlined as follows: • We present ORCA, the first large-scale marine dataset with broad taxonomic coverage, bounding box annotations, and rich instance-level captions. • We conduct an evaluation of 18 models, showing that fine-tuning on ORCA improves performance on localization and captioning tasks. • We demonstrate that dense, domain-specific captions enable accurate object referencing and resolve challenges posed by morphological overlap, where visual cues are ambiguous and misleading. • We show that existing captioning models struggle with instance-level descriptions, often producing coarse, image-level captions instead of region-specific outputs. 2 Related Work Existing marine research . Marine species exhibit high diversity in pose, appearance, and pattern. Robust marine visual understanding can leverage recent algorithms [ 32 , 19 , 83 , 84 ] to advance research, conservation, and industry. Several datasets have been introduced, including MAS3K [ 33 , 32 ] , WildFish [ 88 ] , WildFish++ [ 89 ] , and SUIM [ 21 ] , which improve recognition of marine organisms. However, most of them provide only a limited set of predefined categories without detailed captions, restricting their utility for fine-grained marine analysis and large-scale scientific databases. ORCA addresses this gap by introducing a large-scale dataset covering a broad range of marine species with high-quality annotations (bounding boxes and captions). Object Detection . Object detection is a core computer vision task [ 36 , 55 , 56 ] , involving simultaneous object localization and classification. Conventional one-stage [ 42 , 14 , 54 ] and two-stage [ 55 , 56 , 17 ] detectors rely on fixed predefined category sets, which limit their applicability in marine domains, where species diversity varies greatly across regions. Open-vocabulary object detection (OVOD) [ 75 , 72 , 24 , 66 ] addresses this challenge by extending detection to unseen categories. OVOD commonly leverages large-scale vision–language pre-training [ 51 ] to align visual regions with textual concepts; for instance, RegionCLIP [ 24 ] enhances generalization by matching regional features with natural language. These properties make OVOD particularly promising in marine applications, with the ability to recognize novel and diverse species. Vision–Language Understanding . VLMs [ 40 , 15 , 63 , 87 , 39 , 85 , 30 , 28 , 82 ] have made substantial progress, driven by large-scale datasets such as Visual Genome [ 26 ] , VizWiz [ 16 ] , RefCOCO [ 22 ] , and Objects365 [ 59 ] . These models combine visual encoders [ 12 ] with large language models [ 45 , 46 ] , trained on massive image–text corpora. CLIP [ 51 ] demonstrated strong zero-shot recognition, while BLIP [ 30 , 28 ] advanced multimodal pre-training through frozen encoder–decoder architectures. Collectively, these works provide the foundation for tasks such as image