SpVOS: Efficient Video Object Segmentation With Triple Sparse Convolution

被引:1
|
作者
Lin, Weihao [1 ]
Chen, Tao [1 ]
Yu, Chong [2 ]
机构
[1] Fudan Univ, Sch Informat Sci & Technol, Shanghai 200433, Peoples R China
[2] Fudan Univ, Acad Engn & Technol, Shanghai 200433, Peoples R China
基金
中国国家自然科学基金;
关键词
Video object segmentation; convolutional neural networks; sparse convolution; PROPOSAL GENERATION;
D O I
10.1109/TIP.2023.3327588
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Semi-supervised video object segmentation (Semi-VOS), which requires only annotating the first frame of a video to segment future frames, has received increased attention recently. Among existing Semi-VOS pipelines, the memory-matching-based one is becoming the main research stream, as it can fully utilize the temporal sequence information to obtain high-quality segmentation results. Even though this type of method has achieved promising performance, the overall framework still suffers from heavy computation overhead, mainly caused by the per-frame dense convolution operations between high-resolution feature maps and each kernel filter. Therefore, we propose a sparse baseline of VOS named SpVOS in this work, which develops a novel triple sparse convolution to reduce the computation costs of the overall VOS framework. The designed triple gate, taking full consideration of both spatial and temporal redundancy between adjacent video frames, adaptively makes a triple decision to decide how to apply the sparse convolution on each pixel to control the computation overhead of each layer, while maintaining sufficient discrimination capability to distinguish similar objects and avoid error accumulation. A mixed sparse training strategy, coupled with a designed objective considering the sparsity constraint, is also developed to balance the VOS segmentation performance and computation costs. Experiments are conducted on two mainstream VOS datasets, including DAVIS and Youtube-VOS. Results show that, the proposed SpVOS achieves superior performance over other state-of-the-art sparse methods, and even maintains comparable performance, e.g., an 83.04% (79.29%) overall score on the DAVIS-2017 (Youtube-VOS) validation set, with the typical non-sparse VOS baseline (82.88% for DAVIS-2017 and 80.36% for Youtube-VOS) while saving up to 42% FLOPs, showing its application potential for resource-constrained scenarios.
引用
收藏
页码:5977 / 5991
页数:15
相关论文
共 50 条
  • [21] Breaking the "Object" in Video Object Segmentation
    Tokmakov, Pavel
    Li, Jie
    Gaidon, Adrien
    2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2023, : 22836 - 22845
  • [22] Spatial constraint for efficient semi-supervised video object segmentation
    Chen, Yadang
    Ji, Chuanjun
    Yang, Zhi-Xin
    Wu, Enhua
    COMPUTER VISION AND IMAGE UNDERSTANDING, 2023, 237
  • [23] Efficient foreground object segmentation from video by Probability Weighted Moments
    Umer, Saqib
    Dawood, Hassan
    Yousaf, Muhammad Haroon
    Dawood, Hussain
    Ahmad, Haseeb
    OPTIK, 2021, 229
  • [24] An efficient technique for segmentation of key object(s) from video shots
    Oh, J
    Lee, J
    Vemuri, E
    ITCC 2003: INTERNATIONAL CONFERENCE ON INFORMATION TECHNOLOGY: COMPUTERS AND COMMUNICATIONS, PROCEEDINGS, 2003, : 384 - 388
  • [25] Efficient frame-sequential label propagation for video object segmentation
    Yadang Chen
    Chuanyan Hao
    Wen Wu
    Enhua Wu
    Multimedia Tools and Applications, 2018, 77 : 6117 - 6133
  • [26] Efficient object segmentation using digital matting for MPEG video sequences
    Tsai, YTJ
    Lien, JJJ
    COMPUTER VISION - ACCV 2006, PT II, 2006, 3852 : 591 - 601
  • [27] Efficient, Dense, Object-based Segmentation from RGBD Video
    Ghafarianzadeh, Mahsa
    Blaschko, Matthew B.
    Sibley, Gabe
    2016 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION (ICRA), 2016, : 2310 - 2317
  • [28] Efficient frame-sequential label propagation for video object segmentation
    Chen, Yadang
    Hao, Chuanyan
    Wu, Wen
    Wu, Enhua
    MULTIMEDIA TOOLS AND APPLICATIONS, 2018, 77 (05) : 6117 - 6133
  • [29] Make One-Shot Video Object Segmentation Efficient Again
    Meinhardt, Tim
    Leal-Taixe, Laura
    ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 33, NEURIPS 2020, 2020, 33
  • [30] Boosting Video Object Segmentation via Robust and Efficient Memory Network
    Chen, Yadang
    Zhang, Dingwei
    Zheng, Yuhui
    Yang, Zhi-Xin
    Wu, Enhua
    Zhao, Haixing
    IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 2024, 34 (05) : 3340 - 3352