SyncVIS: Synchronized Video Instance Segmentation
Rongkun Zheng, Xi Chen, Yi Wang, Kun Wang, Yu Qiao, Hengshuang Zhao · 2024
Read SyncVIS: Synchronized Video Instance Segmentation by Rongkun Zheng, Xi Chen, Yi Wang, Kun Wang, Yu Qiao, Hengshuang Zhao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00882.txt.
Introduction
Zheng and six co-authors synchronized video segmentation in this paper from twenty twenty-four. Video instance segmentation must outline and track each object across frames. Transformer methods handle this with queries, but most treat frames and whole clips out of step. The authors trace the resulting failures in hard scenes to that mismatch.
Their framework, SyncVIS, adds video-level query embeddings beside the frame-level ones. One module makes the two levels learn from each other during processing. A second splits long videos into short clips so training stays manageable. On three YouTube benchmarks and the OVIS set, the method reached the top scores. The authors offer synchronization as the fix for complex video.
Catalog record
| CREATOR | Rongkun Zheng, Xi Chen, Yi Wang, Kun Wang, Yu Qiao, Hengshuang Zhao |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00882
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SyncVIS: Synchronized Video Instance Segmentation Rongkun Zheng, Xi Chen, Yi Wang, Kun Wang, Yu Qiao, Hengshuang Zhao Abstract Recent DETR-based methods have advanced the development of Video Instance Segmentation (VIS) through transformers’ efficiency and capability in modeling spatial and temporal information. Despite harvesting remarkable progress, existing works follow asynchronous designs, which model video sequences via either video-level queries only or adopting query-sensitive cascade structures, resulting in difficulties when handling complex and challenging video scenarios. In this work, we analyze the cause of this phenomenon and the limitations of the current solutions, and propose to conduct synchronized modeling via a new framework named SyncVIS. Specifically, SyncVIS explicitly introduces video-level query embeddings and designs two key modules to synchronize video-level query with frame-level query embeddings: a synchronized video-frame modeling paradigm and a synchronized embedding optimization strategy. The former attempts to promote the mutual learning of frame- and video-level embeddings with each other and the latter divides large video sequences into small clips for easier optimization. Extensive experimental evaluations are conducted on the challenging YouTube-VIS 2019 & 2021 & 2022, and OVIS benchmarks, and SyncVIS achieves state-of-the-art results, which demonstrates the effectiveness and generality of the proposed approach. The code is available at https://github.com/rkzheng99/SyncVIS . 1 Introduction Video Instance Segmentation (VIS) is a fundamental while challenging vision task that aims to detect, segment, and track object instances inside videos based on a set of predefined object categories at the same time. With the prosperous video media, VIS has attracted various attention due to its numerous vital applications in areas such as video understanding, video editing, autonomous driving, etc. Benefiting from favorable long-range modeling among frames, query-based offline VIS methods [ 6 , 31 , 15 , 29 , 17 , 36 ] like Mask2Former-VIS [ 6 ] , and SeqFormer [ 31 ] begin to dominate the VIS. Inspired by the object detection method DETR [ 5 ] , they learn a group of queries that can track and segment potential instances simultaneously across the multiple frames of a video. On the other hand, online VIS approaches like IDOL [ 32 ] also exploit the temporal consistency of query embeddings and associate instances via linking the corresponding query embeddings frame by frame. Albeit the success gained by those methods, we find they barely capitalize multi-frame inputs. In practice, the Mask2Former-VIS [ 6 ] would significantly perform worse if more input frames are given during training (evidenced in Fig. 3 ). This is paradoxical to our common sense that more frames could facilitate deep learning models obtaining more motion information of instances. Figure 1: Comparison of video instance segmentation paradigms. Previous methods (left part) like VITA [ 15 ] adopt asynchronous query-sensitive structures to model instance appearances and trajectories. Our model (right part) employs frame and video embeddings in a query-robust synchronous manner, and they synchronize with each other through the transformer decoder to generate the refined video-level query embeddings for the prediction. Also, we employ a synchronized embedding optimization strategy ‘Sync. Optim.’ instead of the classic optimization approach. For this problem, many researchers [ 15 , 31 , 17 ] point out that the video-level queries are vitally hard to track the instances well if receiving many frames in training. That is because the trajectory complexity will increase in polynomials along with the number of frames. Therefore, state-of-the-art methods like SeqFormer [ 31 ] and VITA [ 15 ] usually decouple the trajectory into spatial and temporal dimensions, which are modeled by frame-level and video-level queries, respectively. Specifically, they utilize the frame-level queries to segment each frame independently and then associate these frame-level queries with video-level queries, which are responsible for the final video-level prediction. The well-trained frame-level queries guarantee the quality in the spatial dimension and thus decrease the burden of video queries. However, we argue that two issues remain in these asynchronous designs (as illustrated at the left of Fig. 1 ). First, with the asynchronous structure, the wellness of video-level queries heavily relies on the learning of former frame-level queries, inside which some motion information may be lost because it is an image encoding stage (rather than video encoding), which leads to the sensitivity of queries to the learning quality of former stages. Second, previous works have not solved the bipartite matching among more frames (rather than single frame), and thus the optimization complexity of trajectories remains exorbitant. Both two issues block the further development of query-based methods for video instance segmentation. To this end, we propose to model video and frame queries synchronously with a new framework named SyncVIS to address the above-mentioned issues. Built upon DETR-style structures [ 6 , 32 ] , our SyncVIS has two key components: the synchronized video-frame modeling paradigm and the synchronized embedding optimization strategy. Both designs put effort into unifying the frame- and video-level predictions in synchronization. The synchronized video-frame modeling paradigm makes frame- and video-level embeddings interact with each other in a query-robust parallel manner, rather than a query-sensitive cascade structure. Then the synchronized embedding optimization strategy adds a video-level buffer state to generate more tractable intermediate bipartite matching optimization compared with only frame-level losses. Fig. 1 demonstrates the schematic difference between the asynchronous state-of-the-art method and our synchronous approach. Our model is schematically simple but practically more effective, with exquisite designs as follows. In the synchronized video-frame modeling paradigm, we employ frame and video-level embeddings in the transformer decoder to model object segmentation and tracking synchronously. Specifically, frame-level embeddings are assigned to each sampled frame, and responsible for modeling the appearance of instances, and video-level embeddings are a set of shared instance queries for all sampled frames, which are used to characterize the general motion (In the DETR-style architecture, when video queries are associated with features across time via the decoder, they can effectively model instance-level motion through the cascade structure. In Mask2Former-VIS, the use of video queries alone enables the capture of instance motion). Frame-level embeddings are kept on each frame to attend to instances locally. In each decoder layer, the video-level embeddings are aggregated to refine frame-level embeddings on the corresponding frame. The refined frame-level embeddings, in turn, are aggregated into video-level embeddings. By repeating this synchronization in decoder layers, SyncVIS incorporates the semantics and movement of instances in each frame. In the synchronized embedding optimization strategy, we focus more on video-level bipartite matching. Concretely, we decouple the input video into several clips to synchronize video and frame, and the total number of clips is related to the combinatorial number. Then, we calculate each clip loss independently by video-level bipartite matching, so that video embeddings can maintain their association ability. We evaluate our SyncVIS on four popular VIS benchmarks, including YouTube-VIS 2019 & 2021 & 2022 [ 34 ] , and OVIS-2021 [ 27 ] . The experiments show the effectiveness of our method with significant improvement over the current state-of-the-art methods VITA [ 15 ] , DVIS [ 38 ] , and CTVIS [ 37 ] . Our contributions are as follows: • We analyze the limitations of existing video instance segmentation methods and propose a framework named SyncVIS with synchronized video-frame modeling. It can well characterize instances’ trajectories under complex and challenging video scenarios. • We