Feature Fusion and Metric Learning Network for Zero-Shot Sketch-Based Image Retrieval

被引:2
|
作者
Zhao, Honggang [1 ]
Liu, Mingyue [1 ]
Li, Mingyong [1 ,2 ]
机构
[1] Chongqing Normal Univ, Sch Comp & Informat Sci, Chongqing 401331, Peoples R China
[2] Chongqing Natl Ctr Appl Math, Chongqing 401331, Peoples R China
基金
中国国家自然科学基金;
关键词
sketch retrieval; ResNet-50; attention; metric learning; feature fusion; triplet loss;
D O I
10.3390/e25030502
中图分类号
O4 [物理学];
学科分类号
0702 ;
摘要
Zero-shot sketch-based image retrieval (ZS-SBIR) is an important computer vision problem. The image category in the test phase is a new category that was not visible in the training stage. Because sketches are extremely abstract, the commonly used backbone networks (such as VGG-16 and ResNet-50) cannot handle both sketches and photos. Semantic similarities between the same features in photos and sketches are difficult to reflect in deep models without textual assistance. To solve this problem, we propose a novel and effective feature embedding model called Attention Map Feature Fusion (AMFF). The AMFF model combines the excellent feature extraction capability of the ResNet-50 network with the excellent representation ability of the attention network. By processing the residuals of the ResNet-50 network, the attention map is finally obtained without introducing external semantic knowledge. Most previous approaches treat the ZS-SBIR problem as a classification problem, which ignores the huge domain gap between sketches and photos. This paper proposes an effective method to optimize the entire network, called domain-aware triplets (DAT). Domain feature discrimination and semantic feature embedding can be learned through DAT. In this paper, we also use the classification loss function to stabilize the training process to avoid getting trapped in a local optimum. Compared with the state-of-the-art methods, our method shows a superior performance. For example, on the Tu-berlin dataset, we achieved 61.2 + 1.2% Prec200. On the Sketchy_c100 dataset, we achieved 62.3 + 3.3% mAPall and 75.5 + 1.5% Prec100.
引用
收藏
页数:17
相关论文
共 50 条
  • [31] Deep cross-modal discriminant adversarial learning for zero-shot sketch-based image retrieval
    Jiao, Shichao
    Han, Xie
    Xiong, Fengguang
    Yang, Xiaowen
    Han, Huiyan
    He, Ligang
    Kuang, Liqun
    NEURAL COMPUTING & APPLICATIONS, 2022, 34 (16): : 13469 - 13483
  • [32] Deep cross-modal discriminant adversarial learning for zero-shot sketch-based image retrieval
    Shichao Jiao
    Xie Han
    Fengguang Xiong
    Xiaowen Yang
    Huiyan Han
    Ligang He
    Liqun Kuang
    Neural Computing and Applications, 2022, 34 : 13469 - 13483
  • [33] Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrieval
    Dutta, Anjan
    Akata, Zeynep
    2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019), 2019, : 5084 - 5093
  • [34] Stacked Adversarial Network for Zero-Shot Sketch based Image Retrieval
    Pandey, Anubha
    Mishra, Ashish
    Verma, Vinay Kumar
    Mittal, Anurag
    Murthy, Hema A.
    2020 IEEE WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION (WACV), 2020, : 2529 - 2538
  • [35] Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval
    Liu, Qing
    Xie, Lingxi
    Wang, Huiyu
    Yuile, Alan L.
    2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2019), 2019, : 3661 - 3670
  • [36] Zero-shot sketch-based image retrieval with structure-aware asymmetric disentanglement
    Li, Jiangtong
    Ling, Zhixin
    Niu, Li
    Zhang, Liqing
    COMPUTER VISION AND IMAGE UNDERSTANDING, 2022, 218
  • [37] A Zero-Shot Framework for Sketch Based Image Retrieval
    Yelamarthi, Sasi Kiran
    Reddy, Shiva Krishna
    Mishra, Ashish
    Mittal, Anurag
    COMPUTER VISION - ECCV 2018, PT IV, 2018, 11208 : 316 - 333
  • [38] Domain-aware double attention network for zero-shot sketch-based image retrieval with similarity loss
    Ming Zhu
    Chen Zhao
    Nian Wang
    Feiyang Gu
    Yu Liu
    Xin Li
    The Visual Computer, 2024, 40 : 3091 - 3101
  • [39] Domain-aware double attention network for zero-shot sketch-based image retrieval with similarity loss
    Zhu, Ming
    Zhao, Chen
    Wang, Nian
    Gu, Feiyang
    Liu, Yu
    Li, Xin
    VISUAL COMPUTER, 2024, 40 (05): : 3091 - 3101
  • [40] StyleGuide: Zero-Shot Sketch-Based Image Retrieval Using Style-Guided Image Generation
    Dutta, Titir
    Singh, Anurag
    Biswas, Soma
    IEEE TRANSACTIONS ON MULTIMEDIA, 2021, 23 : 2833 - 2842