End-to-End Supermask Pruning: Learning to Prune Image Captioning Models

被引:13
|
作者
Tan, Jia Huei [1 ]
Chan, Chee Seng [1 ]
Chuah, Joon Huang [2 ]
机构
[1] Univ Malaya, Dept Artificial Intelligence, Ctr Image & Signal Proc CISiP, Kuala Lumpur 50603, Malaysia
[2] Univ Malaya, Dept Elect Engn, Kuala Lumpur 50603, Malaysia
关键词
Image captioning; Deep network compression; Deep learning;
D O I
10.1016/j.patcog.2021.108366
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
With the advancement of deep models, research work on image captioning has led to a remarkable gain in raw performance over the last decade, along with increasing model complexity and computational cost. However, surprisingly works on compression of deep networks for image captioning task has received little to no attention. For the first time in image captioning research, we provide an extensive comparison of various unstructured weight pruning methods on three different popular image captioning architectures, namely Soft-Attention, Up-Down and Object Relation Transformer . Following this, we propose a novel end-to-end weight pruning method that performs gradual sparsification based on weight sensitivity to the training loss. The pruning schemes are then extended with encoder pruning, where we show that conducting both decoder pruning and training simultaneously prior to the encoder pruning provides good overall performance. Empirically, we show that an 80% to 95% sparse network (up to 75% reduction in model size) can either match or outperform its dense counterpart. The code and pre-trained models for Up-Down and Object Relation Transformer that are capable of achieving CIDEr scores > 120 on the MSCOCO dataset but with only 8.7 MB and 14.5 MB in model size (size reduction of 96% and 94% respectively against dense versions) are publicly available at https://github.com/jiahuei/sparse- image-captioning . (c) 2021 Elsevier Ltd. All rights reserved.
引用
收藏
页数:12
相关论文
共 50 条
  • [31] END-TO-END LEARNING OF POLYGONS FOR REMOTE SENSING IMAGE CLASSIFICATION
    Girard, Nicolas
    Tarabalka, Yuliya
    IGARSS 2018 - 2018 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM, 2018, : 2083 - 2086
  • [32] SWINBERT: End-to-End Transformers with Sparse Attention for Video Captioning
    Lin, Kevin
    Li, Linjie
    Lin, Chung-Ching
    Ahmed, Faisal
    Gan, Zhe
    Liu, Zicheng
    Lu, Yumao
    Wang, Lijuan
    2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022), 2022, : 17928 - 17937
  • [33] An End-to-End Image Dehazing Method Based on Deep Learning
    Zhang, Yi
    Huang, Hongbing
    Liu, Junyi
    Fan, Chao
    Wang, Yanyan
    Cai, Qing
    Ruan, Yingying
    Gong, Xiaojin
    2018 3RD INTERNATIONAL CONFERENCE ON COMMUNICATION, IMAGE AND SIGNAL PROCESSING, 2019, 1169
  • [34] Pruning DETR: efficient end-to-end object detection with sparse structured pruning
    Huaiyuan Sun
    Shuili Zhang
    Xve Tian
    Yuanyuan Zou
    Signal, Image and Video Processing, 2024, 18 : 129 - 135
  • [35] Pruning DETR: efficient end-to-end object detection with sparse structured pruning
    Sun, Huaiyuan
    Zhang, Shuili
    Tian, Xve
    Zou, Yuanyuan
    SIGNAL IMAGE AND VIDEO PROCESSING, 2024, 18 (01) : 129 - 135
  • [36] Online Continual Learning of End-to-End Speech Recognition Models
    Yang, Muqiao
    Lane, Ian
    Watanabe, Shinji
    INTERSPEECH 2022, 2022, : 2668 - 2672
  • [37] END-TO-END SPATIALLY-CONSTRAINED MULTI-PERSPECTIVE FINE-GRAINED IMAGE CAPTIONING
    Zhang, Yifan
    Lin, Chunzhen
    Cao, Donglin
    Lin, Dazhen
    2024 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING, ICASSP 2024, 2024, : 3360 - 3364
  • [38] Defining the "to" in end-to-end models
    Mitra, Aditee
    Davis, Cabell
    PROGRESS IN OCEANOGRAPHY, 2010, 84 (1-2) : 39 - 42
  • [39] Compression of End-to-End Models
    Pang, Ruoming
    Sainath, Tara N.
    Prabhavalkar, Rohit
    Gupta, Suyog
    Wu, Yonghui
    Zhang, Shuyuan
    Chiu, Chung-cheng
    19TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH COMMUNICATION ASSOCIATION (INTERSPEECH 2018), VOLS 1-6: SPEECH RESEARCH FOR EMERGING MARKETS IN MULTILINGUAL SOCIETIES, 2018, : 27 - 31
  • [40] Federated End-to-End Unrolled Models for Magnetic Resonance Image Reconstruction
    Levac, Brett R.
    Arvinte, Marius
    Tamir, Jonathan I.
    BIOENGINEERING-BASEL, 2023, 10 (03):