AUTV: Creating Underwater Video Datasets with Pixel-wise Annotations
Quang Trung Truong Wong Yuk Kwan Duc Thanh Nguyen Binh-Son Hua Sai-Kit Yeung · 2025
Read AUTV: Creating Underwater Video Datasets with Pixel-wise Annotations by Quang Trung Truong Wong Yuk Kwan Duc Thanh Nguyen Binh-Son Hua Sai-Kit Yeung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12828.txt.
Introduction
Quang Trung Truong and four co-authors manufacture underwater video datasets because real ones are too hard to label, in a 2025 paper. The sea defeats annotation: water distorts, cameras drift, and creatures refuse to sit still. Computer vision for marine monitoring needs training data the field cannot easily supply.
Their framework synthesizes marine videos with exact, pixel-level labels. They built two collections with it. One gathers two thousand real videos paired with text and rich notes on appearance, lighting, camera properties, and animal behavior. The other fabricates ten thousand synthetic clips with segmentation masks tracing each marine object. The synthetic set demonstrably improves downstream tasks like filling occluded regions and tracking objects through video. For ocean science, synthetic data may be the only scalable route to capable vision models.
Catalog record
| CREATOR | Quang Trung Truong Wong Yuk Kwan Duc Thanh Nguyen Binh-Son Hua Sai-Kit Yeung |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-12828
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AUTV: Creating Underwater Video Datasets with Pixel-wise Annotations Quang Trung Truong Wong Yuk Kwan Duc Thanh Nguyen Binh-Son Hua Sai-Kit Yeung Abstract Underwater video analysis, hampered by the dynamic marine environment and camera motion, remains a challenging task in computer vision. Existing training-free video generation techniques, learning motion dynamics on the frame-by-frame basis, often produce poor results with noticeable motion interruptions and misaligments. To address these issues, we propose AUTV , a framework for synthesizing marine video data with pixel-wise annotations. We demonstrate the effectiveness of this framework by constructing two video datasets, namely UTV , a real-world dataset comprising 2,000 video-text pairs, and SUTV , a synthetic video dataset including 10,000 videos with segmentation masks for marine objects. UTV provides diverse underwater videos with comprehensive annotations including appearance, texture, camera intrinsics, lighting, and animal behavior. SUTV can be used to improve underwater downstream tasks, which are demonstrated in video inpainting and video object segmentation. Figure 1 : Given masks of the first video frame and a text, or a text, our method synthesizes a high-fidelity video 1 Introduction Approximately 75% of the Earth’s surface (or 362 million km^{2} , equivalently) is dominated by oceans and major seas. These vast bodies of water are integral to climate regulation and serve as a primary source of oxygen for the planet. However, marine species are considerably less well-documented compared with land species. Almost 89% of marine protected areas are under-explored [ 18 ] and only 200 marine areas are recorded in the World Database on Protected Areas [ 22 ] . The underwater computer vision domain aims to interpret marine imagery and video footage. The literature has shown a large body of research methods devoted to underwater video analysis. For instance, underwater instance segmentation is studied in [ 13 , 14 ] . FishNet [ 11 ] provides a large-scale dataset of fish imagery, supporting various vision tasks such as fish detection and recognition. IOCFormer [ 30 ] addresses the underwater object counting problem. MarineInst [ 48 ] leverages marine data annotation by utilizing both human-annotated and model-generated instance masks for training of instance segmentation models. Depth estimation and underwater image restoration are studied in [ 34 ] . Figure 2 : Generated video object masks (first row) and corresponding video frames (second row) by applying SegGen [ 45 ] in a frame-wise basis. As show, there is a huge discrepancy between the generated masks and video frames, due to a lack of integration of motion during the synthesis process. To facilitate underwater computer vision research, marine datasets have been created. For instance, Marine-tree [ 4 ] includes 160k annotated images with 60 object classes organized in a tree structure. UTB180 [ 1 ] contains 180 marine videos for underwater object tracking. MVK [ 31 ] consists of 44,330 text-image pairs designed for marine retrieval applications. CBIL [ 40 ] leverages video-based motion prior to guide a model to learn movement patterns for adversarial imitation. However, CBIL heavily relies on training segmentation masks for motion synthesis. In this paper, we propose a workflow that performs simultaneously video generation and object mask annotation from a textual description or a combination of a textual description and initial masks (see Figure 1 ). This capability is enabled by text-to-video (T2V) and object segmentation techniques. As shown in the literature, T2V presents a promising approach to creating abundant video content based on textual descriptions. This technology not only improves the richness of marine data, but also supports marine biology studies such as marine species identification and habitat evaluation. A straightforward approach is to leverage text-to-image and mask-to-image methods to video generation on a frame-wise basis. For instance, SegGen [ 45 ] generates object masks for individual video frames using a text-to-image model, fine-tuned on an instance segmentation dataset (e.g., COCO [ 15 ] ). These masks are subsequently used as inputs to a mask-to-image model for video frame generation. However, we observed challenges in this approach. First, it is difficult to ensure temporal consistency in object masks trained with limited data while aligning the masks with generated video frames (see Figure 2 ). Employing video temporal smoothing techniques could be a treatment to enhance the temporal consistency. However, these techniques alter the generated video content, leading to discrepancies and misalignments between the object masks and the generated video frames. Secondly, video mask generation techniques, trained with video object segmentation (VOS) datasets, such as Ref-YouTube-VOS [ 25 ] , Ref-DAVIS17 [ 12 ] , MeViS [ 7 ] , often fail to achieve temporal consistency in the mask generation results as the referring datasets are captured at different frame rates, and hence diverse in motion flow and speed. To enforce the alignment between the generated video frames and object masks while ensuring the smoothness of the motion flow in the generated contents, we fine-tune an off-the-shelf video diffusion model pre-trained on a large-scale dataset, e.g., WebVid-10M [ 2 ] , to generate the video content and then apply a conditional segmentation model, e.g., SAM2 [ 20 ] , with initial masks in the first frame as the condition, to generate object masks in the following frames. In our method, we specifically focus on effectively incorporating motion dynamics into the image-to-video synthesis. In summary, we make the following contributions. • We propose AUTV , a T2V framework for synthesizing marine videos and pixel-wise annotations (object masks). This framework aims at automated data synthesis at scale. • We collect and annotate a real-world video-text pair dataset, namely UTV , for marine research. This dataset is used to fine-tune a generic T2V model for marine video data synthesis. • We introduce a synthetic marine video dataset with pixel-wise object annotations using our proposed framework. We name this dataset SUTV . • We demonstrate the usefulness of the SUTV dataset in two downstream applications: video inpainting and video object segmentation. 2 Related work Text-to-Video (T2V) Diffusion-based T2V generation models trained on large-scale datasets, e.g., WebVid-10M [ 2 ] , LAION-400M [ 24 ] , have made significant advances in the field. For instance, Make-A-Video [ 27 ] employed a two-step approach that aligns text-to-image (T2I) and T2V tasks by training a joint foundational model. Following this research direction, Movie Gen [ 19 ] was built on the autoencoder architecture in [ 23 ] , and enhanced by incorporating temporal convolutions with conventional spatial convolutions. The advantage of this method is its ability to encode videos of varying lengths. VideoLDM [ 3 ] and ModelScopeT2V [ 35 ] extended the 2D-UNet architecture in the stable diffusion model [ 23 ] to 3D-UNet by including temporal layers. This approach was then adopted in [ 37 , 36 ] . VideoCrafter1 [ 5 ] is an open-source high-resolution video generator, built on the latent diffusion model in [ 8 ] , and offering both T2V and image-to-video (I2V) functionalities. Latte [ 16 ] applied a series of transformer blocks to the spatio-temporal tokens of training video sequences to learn the distribution of the video data in a latent space. (a) (b) Figure 3 : Overview of our AUTV framework. (a) Video generation pipeline. (b) Image2Video module built on a video diffusion model (VDM), i.e.,