iSeeBetter: Spatio-temporal video super-resolution using recurrent generative back-projection networks

被引:16
|
作者
Chadha, Aman [1 ]
Britto, John [2 ]
Roja, M. Mani [3 ]
机构
[1] Stanford Univ, Dept Comp Sci, 450 Serra Mall, Stanford, CA 94305 USA
[2] Univ Massachusetts, Dept Comp Sci, Amherst, MA 01003 USA
[3] Univ Mumbai, Dept Elect & Telecommun Engn, Mumbai 400032, Maharashtra, India
关键词
super resolution; video upscaling; frame recurrence; optical flow; generative adversarial networks; convolutional neural networks; IMAGE; RESOLUTION;
D O I
10.1007/s41095-020-0175-7
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
Recently, learning-based models have enhanced the performance of single-image super-resolution (SISR). However, applying SISR successively to each video frame leads to a lack of temporal coherency. Convolutional neural networks (CNNs) outperform traditional approaches in terms of image quality metrics such as peak signal to noise ratio (PSNR) and structural similarity (SSIM). On the other hand, generative adversarial networks (GANs) offer a competitive advantage by being able to mitigate the issue of a lack of finer texture details, usually seen with CNNs when super-resolving at large upscaling factors. We present iSeeBetter, a novel GAN-based spatio-temporal approach to video super-resolution (VSR) that renders temporally consistent super-resolution videos. iSeeBetter extracts spatial and temporal information from the current and neighboring frames using the concept of recurrent back-projection networks as its generator. Furthermore, to improve the "naturality" of the super-resolved output while eliminating artifacts seen with traditional algorithms, we utilize the discriminator from super-resolution generative adversarial network. Although mean squared error (MSE) as a primary loss-minimization objective improves PSNR/SSIM, these metrics may not capture fine details in the image resulting in misrepresentation of perceptual quality. To address this, we use a four-fold (MSE, perceptual, adversarial, and total-variation loss function. Our results demonstrate that iSeeBetter offers superior VSR fidelity and surpasses state-of-the-art performance.
引用
收藏
页码:307 / 317
页数:11
相关论文
共 50 条
  • [11] Progressive back-projection networks for large-scale super-resolution
    Yang, Ye
    Fan, Cien
    Tian, Sheng
    Guo, Yang
    Liu, Lingzhi
    Wu, Minyuan
    JOURNAL OF ELECTRONIC IMAGING, 2019, 28 (03)
  • [12] Super-Resolution Reconstruction for Spatio-Temporal Resolution Enhancement of Video Sequences
    Haseyama, Miki
    Izumi, Daisuke
    Takizawa, Makoto
    IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, 2012, E95D (09): : 2355 - 2358
  • [13] Video Image Super-resolution Restoration Based on Iterative Back-Projection Algorithm
    Wan, Baikun
    Meng, Lin
    Ming, Dong
    Qi, Hongzhi
    Hu, Yong
    Luk, K. D. K.
    2009 IEEE INTERNATIONAL CONFERENCE ON COMPUTATIONAL INTELLIGENCE FOR MEASUREMENT SYSTEMS AND APPLICATIONS, 2009, : 46 - +
  • [14] Video super-resolution based on a spatio-temporal matching network
    Zhu, Xiaobin
    Li, Zhuangzi
    Lou, Jungang
    Shen, Qing
    PATTERN RECOGNITION, 2021, 110
  • [15] Residual Invertible Spatio-Temporal Network for Video Super-Resolution
    Zhu, Xiaobin
    Li, Zhuangzi
    Zhang, Xiao-Yu
    Li, Changsheng
    Liu, Yaqi
    Xue, Ziyu
    THIRTY-THIRD AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE / THIRTY-FIRST INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE / NINTH AAAI SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2019, : 5981 - 5988
  • [16] Grouped Spatio-Temporal Alignment Network for Video Super-Resolution
    Lu, Mingxuan
    Zhang, Peng
    IEEE SIGNAL PROCESSING LETTERS, 2022, 29 : 2193 - 2197
  • [17] Video Super-Resolution via a Spatio-Temporal Alignment Network
    Wen, Weilei
    Ren, Wenqi
    Shi, Yinghuan
    Nie, Yunfeng
    Zhang, Jingang
    Cao, Xiaochun
    IEEE TRANSACTIONS ON IMAGE PROCESSING, 2022, 31 : 1761 - 1773
  • [18] Fast Spatio-Temporal Residual Network for Video Super-Resolution
    Li, Sheng
    He, Fengxiang
    Du, Bo
    Zhang, Lefei
    Xu, Yonghao
    Tao, Dacheng
    2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019), 2019, : 10514 - 10523
  • [19] SUPER-RESOLUTION BASED ON BACK-PROJECTION OF INTERPOLATED IMAGE
    Kiatpapan, Sawiya
    Yamaguchi, Takuro
    Ikehara, Masaaki
    2019 12TH INTERNATIONAL CONFERENCE ON ADVANCED TECHNOLOGIES FOR COMMUNICATIONS (ATC 2019), 2019, : 302 - 307
  • [20] Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation
    Caballero, Jose
    Ledig, Christian
    Aitken, Andrew
    Acosta, Alejandro
    Totz, Johannes
    Wang, Zehan
    Shi, Wenzhe
    30TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2017), 2017, : 2848 - 2857