Spatio-Temporal Inference Transformer Network for Video Inpainting

被引:0
|
作者
Tudavekar, Gajanan [1 ,2 ]
Saraf, Santosh S. [2 ]
Patil, Sanjay R. [3 ]
机构
[1] Angadi Inst Technol & Management, Dept Elect & Commun Engn, Belagavi 590009, Karnataka, India
[2] KLS Gogte Inst Technol, Dept Elect & Commun Engn, Belagavi 590008, Karnataka, India
[3] Dnyanshree Inst Engn & Technol, Dept Elect & Telecommun Engn, Sajjangad Rd, Satara 415013, Maharashtra, India
关键词
Image inpainting; video inpainting; Transformer Network; deep learning; IMAGE QUALITY ASSESSMENT; OBJECT REMOVAL;
D O I
10.1142/S0219467823500079
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
Video inpainting aims to complete in a visually pleasing way the missing regions in video frames. Video inpainting is an exciting task due to the variety of motions across different frames. The existing methods usually use attention models to inpaint videos by seeking the damaged content from other frames. Nevertheless, these methods suffer due to irregular attention weight from spatio-temporal dimensions, thus giving rise to artifacts in the inpainted video. To overcome the above problem, Spatio-Temporal Inference Transformer Network (STITN) has been proposed. The STITN aligns the frames to be inpainted and concurrently inpaints all the frames, and a spatio-temporal adversarial loss function improves the STITN. Our method performs considerably better than the existing deep learning approaches in quantitative and qualitative evaluation.
引用
收藏
页数:16
相关论文
共 50 条
  • [1] Spatio-Temporal Transformer Network for Video Restoration
    Kim, Tae Hyun
    Sajjadi, Mehdi S. M.
    Hirsch, Michael
    Schoelkopf, Bernhard
    COMPUTER VISION - ECCV 2018, PT III, 2018, 11207 : 111 - 127
  • [2] DSTA-Net: Deformable Spatio-Temporal Attention Network for Video Inpainting
    Liu, Tongxing
    Qiu, Guoxin
    Xuan, Hanyu
    IEEE SIGNAL PROCESSING LETTERS, 2025, 32 : 771 - 775
  • [3] Video Inpainting Algorithm Using Spatio-Temporal Consistency
    Lee, Sang-Heon
    Lee, Soon-Young
    Heu, Jun-Hee
    Kim, Chang-Su
    Lee, Sang-Uk
    COMPUTATIONAL IMAGING VII, 2009, 7246
  • [4] A Spatio-temporal Inpainting Method for Kinect Depth Video
    Zhang, Dongdong
    Yao, Ye
    Zang, Di
    Chen, Yanyu
    2013 IEEE INTERNATIONAL CONFERENCE ON SIGNAL AND IMAGE PROCESSING APPLICATIONS (IEEE ICSIPA 2013), 2013, : 67 - 70
  • [5] SPATIO-TEMPORAL BINARY VIDEO INPAINTING VIA THRESHOLD DYNAMICS
    Oliver, M.
    Palomares, R. P.
    Ballester, C.
    Haro, G.
    2017 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2017, : 1822 - 1826
  • [6] Spatio-temporal texture synthesis and image inpainting for video applications
    Kumar, S
    Biswas, M
    Belongie, SJ
    Nguyen, TQ
    2005 INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), VOLS 1-5, 2005, : 2317 - 2320
  • [7] Spatio-Temporal Transformer Network for Weather Forecasting
    Ji, Junzhong
    He, Jing
    Lei, Minglong
    Wang, Muhua
    Tang, Wei
    IEEE TRANSACTIONS ON BIG DATA, 2025, 11 (02) : 372 - 387
  • [8] Transformer with Spatio-Temporal Representation for Video Anomaly Detection
    Sun, Xiaohu
    Chen, Jinyi
    Shen, Xulin
    Li, Hongjun
    STRUCTURAL, SYNTACTIC, AND STATISTICAL PATTERN RECOGNITION, S+SSPR 2022, 2022, 13813 : 213 - 222
  • [9] Parallel Spatio-Temporal Attention Transformer for Video Frame Interpolation
    Ning, Xin
    Cai, Feifan
    Li, Yuhang
    Ding, Youdong
    ELECTRONICS, 2024, 13 (10)
  • [10] Spatio-Temporal Graph Convolution Transformer for Video Question Answering
    Tang, Jiahao
    Hu, Jianguo
    Huang, Wenjun
    Shen, Shengzhi
    Pan, Jiakai
    Wang, Deming
    Ding, Yanyu
    IEEE ACCESS, 2024, 12 : 131664 - 131680