Feature Fusion and Metric Learning Network for Zero-Shot Sketch-Based Image Retrieval

被引:2
|
作者
Zhao, Honggang [1 ]
Liu, Mingyue [1 ]
Li, Mingyong [1 ,2 ]
机构
[1] Chongqing Normal Univ, Sch Comp & Informat Sci, Chongqing 401331, Peoples R China
[2] Chongqing Natl Ctr Appl Math, Chongqing 401331, Peoples R China
基金
中国国家自然科学基金;
关键词
sketch retrieval; ResNet-50; attention; metric learning; feature fusion; triplet loss;
D O I
10.3390/e25030502
中图分类号
O4 [物理学];
学科分类号
0702 ;
摘要
Zero-shot sketch-based image retrieval (ZS-SBIR) is an important computer vision problem. The image category in the test phase is a new category that was not visible in the training stage. Because sketches are extremely abstract, the commonly used backbone networks (such as VGG-16 and ResNet-50) cannot handle both sketches and photos. Semantic similarities between the same features in photos and sketches are difficult to reflect in deep models without textual assistance. To solve this problem, we propose a novel and effective feature embedding model called Attention Map Feature Fusion (AMFF). The AMFF model combines the excellent feature extraction capability of the ResNet-50 network with the excellent representation ability of the attention network. By processing the residuals of the ResNet-50 network, the attention map is finally obtained without introducing external semantic knowledge. Most previous approaches treat the ZS-SBIR problem as a classification problem, which ignores the huge domain gap between sketches and photos. This paper proposes an effective method to optimize the entire network, called domain-aware triplets (DAT). Domain feature discrimination and semantic feature embedding can be learned through DAT. In this paper, we also use the classification loss function to stabilize the training process to avoid getting trapped in a local optimum. Compared with the state-of-the-art methods, our method shows a superior performance. For example, on the Tu-berlin dataset, we achieved 61.2 + 1.2% Prec200. On the Sketchy_c100 dataset, we achieved 62.3 + 3.3% mAPall and 75.5 + 1.5% Prec100.
引用
收藏
页数:17
相关论文
共 50 条
  • [1] Attention map feature fusion network for Zero-Shot Sketch-based Image Retrieval
    Zhao, Honggang
    Liu, Mingyue
    Lin, Yinghua
    Li, Mingyong
    2023 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS, IJCNN, 2023,
  • [2] Energy-Guided Feature Fusion for Zero-Shot Sketch-Based Image Retrieval
    Ren, Hao
    Zheng, Ziqiang
    Lu, Hong
    NEURAL PROCESSING LETTERS, 2022, 54 (06) : 5711 - 5720
  • [3] Energy-Guided Feature Fusion for Zero-Shot Sketch-Based Image Retrieval
    Hao Ren
    Ziqiang Zheng
    Hong Lu
    Neural Processing Letters, 2022, 54 : 5711 - 5720
  • [4] Transferable Coupled Network for Zero-Shot Sketch-Based Image Retrieval
    Wang, Hao
    Deng, Cheng
    Liu, Tongliang
    Tao, Dacheng
    IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2022, 44 (12) : 9181 - 9194
  • [5] Contour detection network for zero-shot sketch-based image retrieval
    Zhang, Qing
    Zhang, Jing
    Su, Xiangdong
    Bao, Feilong
    Gao, Guanglai
    COMPLEX & INTELLIGENT SYSTEMS, 2023, 9 (06) : 6781 - 6795
  • [6] Contour detection network for zero-shot sketch-based image retrieval
    Qing Zhang
    Jing Zhang
    Xiangdong Su
    Feilong Bao
    Guanglai Gao
    Complex & Intelligent Systems, 2023, 9 : 6781 - 6795
  • [7] Zero-shot sketch-based image retrieval via adaptive relation-aware metric learning
    Liu, Yang
    Dang, Yuhao
    Gao, Xinbo
    Han, Jungong
    Shao, Ling
    PATTERN RECOGNITION, 2024, 152
  • [8] Domain-Smoothing Network for Zero-Shot Sketch-Based Image Retrieval
    Wang, Zhipeng
    Wang, Hao
    Yan, Jiexi
    Wu, Aming
    Deng, Cheng
    PROCEEDINGS OF THE THIRTIETH INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, IJCAI 2021, 2021, : 1143 - 1149
  • [9] Zero-Shot Sketch-Based Image Retrieval via Graph Convolution Network
    Zhang, Zhaolong
    Zhang, Yuejie
    Feng, Rui
    Zhang, Tao
    Fan, Weiguo
    THIRTY-FOURTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, THE THIRTY-SECOND INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE AND THE TENTH AAAI SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2020, 34 : 12943 - 12950
  • [10] Generative Model for Zero-Shot Sketch-Based Image Retrieval
    Verma, Vinay Kumar
    Mishra, Aakansha
    Mishra, Ashish
    Rai, Piyush
    2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW 2019), 2019, : 704 - 713