Feature Fusion and Metric Learning Network for Zero-Shot Sketch-Based Image Retrieval

被引:2
|
作者
Zhao, Honggang [1 ]
Liu, Mingyue [1 ]
Li, Mingyong [1 ,2 ]
机构
[1] Chongqing Normal Univ, Sch Comp & Informat Sci, Chongqing 401331, Peoples R China
[2] Chongqing Natl Ctr Appl Math, Chongqing 401331, Peoples R China
基金
中国国家自然科学基金;
关键词
sketch retrieval; ResNet-50; attention; metric learning; feature fusion; triplet loss;
D O I
10.3390/e25030502
中图分类号
O4 [物理学];
学科分类号
0702 ;
摘要
Zero-shot sketch-based image retrieval (ZS-SBIR) is an important computer vision problem. The image category in the test phase is a new category that was not visible in the training stage. Because sketches are extremely abstract, the commonly used backbone networks (such as VGG-16 and ResNet-50) cannot handle both sketches and photos. Semantic similarities between the same features in photos and sketches are difficult to reflect in deep models without textual assistance. To solve this problem, we propose a novel and effective feature embedding model called Attention Map Feature Fusion (AMFF). The AMFF model combines the excellent feature extraction capability of the ResNet-50 network with the excellent representation ability of the attention network. By processing the residuals of the ResNet-50 network, the attention map is finally obtained without introducing external semantic knowledge. Most previous approaches treat the ZS-SBIR problem as a classification problem, which ignores the huge domain gap between sketches and photos. This paper proposes an effective method to optimize the entire network, called domain-aware triplets (DAT). Domain feature discrimination and semantic feature embedding can be learned through DAT. In this paper, we also use the classification loss function to stabilize the training process to avoid getting trapped in a local optimum. Compared with the state-of-the-art methods, our method shows a superior performance. For example, on the Tu-berlin dataset, we achieved 61.2 + 1.2% Prec200. On the Sketchy_c100 dataset, we achieved 62.3 + 3.3% mAPall and 75.5 + 1.5% Prec100.
引用
收藏
页数:17
相关论文
共 50 条
  • [21] Zero-Shot Sketch-Based Image Retrieval with Hybrid Information Fusion and Sample Relationship Modeling
    Wu, Weijie
    Li, Jun
    Wu, Zhijian
    Xu, Jianhua
    MULTIMEDIA MODELING, MMM 2025, PT IV, 2025, 15523 : 337 - 350
  • [22] Cross-Domain Feature Semantic Calibration for Zero-Shot Sketch-Based Image Retrieval
    He, Xuewan
    Wang, Jielei
    Xia, Qianxin
    Lu, Guoming
    Tang, Yuan
    Lu, Hongxia
    2024 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO, ICME 2024, 2024,
  • [23] Augmented Multimodality Fusion for Generalized Zero-Shot Sketch-Based Visual Retrieval
    Jing, Taotao
    Xia, Haifeng
    Hamm, Jihun
    Ding, Zhengming
    IEEE TRANSACTIONS ON IMAGE PROCESSING, 2022, 31 : 3657 - 3668
  • [24] Domain disentanglement and fusion based on hyperbolic neural networks for zero-shot sketch-based image retrieval
    Zhang, Qing
    Zhang, Jing
    Su, Xiangdong
    Wang, Yonghe
    Bao, Feilong
    Gao, Guanglai
    INFORMATION PROCESSING & MANAGEMENT, 2025, 62 (01)
  • [25] Progressive Cross-Modal Semantic Network for Zero-Shot Sketch-Based Image Retrieval
    Deng, Cheng
    Xu, Xinxun
    Wang, Hao
    Yang, Muli
    Tao, Dacheng
    IEEE TRANSACTIONS ON IMAGE PROCESSING, 2020, 29 : 8892 - 8902
  • [26] XPNet: Cross-Domain Prototypical Network for Zero-Shot Sketch-Based Image Retrieval
    Li, Mingkang
    Qi, Yonggang
    PATTERN RECOGNITION AND COMPUTER VISION, PT I, PRCV 2022, 2022, 13534 : 394 - 410
  • [27] Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learning
    Singha, Mainak
    Jha, Ankit
    Gupta, Divyam
    Singla, Pranav
    Banerjee, Biplab
    COMPUTER VISION - ECCV 2024, PT XXIV, 2025, 15082 : 1 - 19
  • [28] Zero-shot Sketch-based Image Retrieval with Adaptive Balanced Discriminability and Generalizability
    Tian, Jialin
    Xu, Xing
    Cao, Zuo
    Zhang, Gong
    Shen, Fumin
    Yang, Yang
    PROCEEDINGS OF THE 2023 ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA RETRIEVAL, ICMR 2023, 2023, : 407 - 415
  • [29] Cross-Domain Alignment for Zero-Shot Sketch-Based Image Retrieval
    Wang, Xu
    Peng, Dezhong
    Hu, Peng
    Gong, Yunhong
    Chen, Yong
    IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 2023, 33 (11) : 7024 - 7035
  • [30] Asymmetric Mutual Alignment for Unsupervised Zero-Shot Sketch-Based Image Retrieval
    Yin, Zhihui
    Yan, Jiexi
    Xu, Chenghao
    Deng, Cheng
    THIRTY-EIGHTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, VOL 38 NO 15, 2024, : 16504 - 16512