Conversational Image Search: A Sketch-based Approach

被引:1
|
作者
Braghis, Daniel D. [1 ]
Liu, Haiming [1 ]
机构
[1] Univ Southampton, Sch Elect & Comp Sci, Southampton, Hants, England
来源
PROCEEDINGS OF THE 4TH ANNUAL ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA RETRIEVAL, ICMR 2024 | 2024年
关键词
Conversational product search; natural language feedback; multi-modal interaction; sketch-based image retrieval; Stable Diffusion; ControlNet; GPT Assistant;
D O I
10.1145/3652583.3657594
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Conversational image search has emerged as a progressive step beyond traditional keyword-based methodologies, which addresses challenges in human-computer interaction during the information retrieval process. This paper introduces a demonstration called DoodleShoper, a forward-thinking conversational image search assistant centered around sketching, specifically tailored for online product searches. It underscores the importance of visual diversity, often eluding verbal expression while highlighting the efficacy of a sketch-based approach in enhancing user interaction. The proposed modular architecture integrates a state-of-the-art Language Model with advanced Stable Diffusion technologies in the image generation field to offer users a more intuitive and precise conversational search experience. Unlike most conventional methods that directly align prompts or sketches with images, our approach leverages a generative model to produce an intermediate search outcome. This strategic shift streamlines the search process from a zero-shot query - where the query directly corresponds to an image - to a reverse image search task, facilitating the discovery of similar images through multimodal interaction. The implemented demonstration involves refining and expanding the application to diverse user information needs and preferences, including exploring the potential of utilising sketches as an alternative or complementary search environment, a novel concept rooted in current research.
引用
收藏
页码:1265 / 1269
页数:5
相关论文
共 50 条
  • [21] Sketch-based Image Retrieval by Using Saliency
    Pan, Da
    Shi, Ping
    Li, Cuiying
    2014 11TH INTERNATIONAL CONFERENCE ON FUZZY SYSTEMS AND KNOWLEDGE DISCOVERY (FSKD), 2014, : 825 - 829
  • [22] Quadruplet Networks for Sketch-Based Image Retrieval
    Seddati, Omar
    Dupont, Stephane
    Mahmoudi, Said
    PROCEEDINGS OF THE 2017 ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA RETRIEVAL (ICMR'17), 2017, : 189 - 196
  • [23] A Novel Visual-Region-Descriptor-based Approach to Sketch-based Image Retrieval
    Jin, Cheng
    Wang, Zheming
    Zhang, Tianhao
    Zhu, Qinen
    Zhang, Yuejie
    ICMR'15: PROCEEDINGS OF THE 2015 ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA RETRIEVAL, 2015, : 267 - 274
  • [24] CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing
    Xiao, Chufeng
    Fu, Hongbo
    COMPUTER GRAPHICS FORUM, 2024, 43 (07)
  • [25] Concealment of visual effects of image transmission errors by a sketch-based recovery approach
    Atzori, L
    De Natale, FGB
    1998 INTERNATIONAL CONFERENCE ON IMAGE PROCESSING - PROCEEDINGS, VOL 3, 1998, : 482 - 486
  • [26] How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?
    Koley, Subhadeep
    Bhunia, Ayan Kumar
    Sain, Aneeshan
    Chowdhury, Pinaki Nath
    Xiang, Tao
    Song, Yi-Zhe
    2024 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2024, : 16859 - 16869
  • [27] Better freehand sketch synthesis for sketch-based image retrieval: Beyond image edges
    Zhang, Xianlin
    Li, Xueming
    Li, Xuewei
    Shen, Mengling
    NEUROCOMPUTING, 2018, 322 : 38 - 46
  • [28] Sketch-Based Similarity Search for Collaborative Feature Maps
    Leibetseder, Andreas
    Kletz, Sabrina
    Schoeffmann, Klaus
    MULTIMEDIA MODELING, MMM 2018, PT II, 2018, 10705 : 425 - 430
  • [29] A sketch-based approach to human body modelling
    Mao, Chen
    Qjn, Sheng Feng
    Wright, David
    COMPUTERS & GRAPHICS-UK, 2009, 33 (04): : 521 - 541
  • [30] MindCamera: Interactive Sketch-Based Image Retrieval and Synthesis
    Wang, Jingyu
    Zhao, Yu
    Qi, Qi
    Huo, Qiming
    Zou, Jian
    Ge, Ce
    Liao, Jianxin
    IEEE ACCESS, 2018, 6 : 3765 - 3773