Can LSH (locality-sensitive hashing) be replaced by neural network?

被引:1
|
作者
Liu, Renyang [1 ,3 ]
Zhao, Jun [4 ]
Chu, Xing [2 ,3 ]
Liang, Yu [2 ,3 ]
Zhou, Wei [2 ,3 ]
He, Jing [2 ,3 ]
机构
[1] Yunnan Univ, Sch Informat Sci & Engn, Kunming 650500, Yunnan, Peoples R China
[2] Yunnan Univ, Sch Software, Kunming 650500, Yunnan, Peoples R China
[3] Yunnan Univ, Engn Res Ctr Cyberspace, Kunming 650500, Yunnan, Peoples R China
[4] Didi Chuxing, Beijing 100000, Peoples R China
基金
中国国家自然科学基金;
关键词
Learned index; Deep learning; Locality-sensitive hashing; KNN; CODES;
D O I
10.1007/s00500-023-09402-3
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
With the rapid development of GPU (graphics processing unit) technologies and neural networks, we can explore more appropriate data structures and algorithms. Recent progress shows that neural networks can partly replace traditional data structures. In this paper, we proposed a novel DNN (deep neural network)-based learned locality-sensitive hashing, called LLSH, to efficiently and flexibly map high-dimensional data to low-dimensional space. LLSH replaces the traditional LSH (locality-sensitive hashing) function families with parallel multi-layer neural networks, which reduces the time and memory consumption and guarantees query accuracy simultaneously. The proposed LLSH demonstrate the feasibility of replacing the hash index with learning-based neural networks and open a new door for developers to design and configure data organization more accurately to improve information-searching performance. Extensive experiments on different types of data sets show the superiority of the proposed method in query accuracy, time consumption, and memory usage.
引用
收藏
页码:887 / 902
页数:16
相关论文
共 50 条
  • [1] Can LSH (locality-sensitive hashing) be replaced by neural network?
    Renyang Liu
    Jun Zhao
    Xing Chu
    Yu Liang
    Wei Zhou
    Jing He
    Soft Computing, 2024, 28 : 1041 - 1053
  • [2] BCH-LSH: a new scheme of locality-sensitive hashing
    Ma, Yuena
    Feng, Xiaoyi
    Liu, Yang
    Li, Shuhong
    IET IMAGE PROCESSING, 2018, 12 (06) : 850 - 855
  • [3] In Defense of Locality-Sensitive Hashing
    Ding, Kun
    Huo, Chunlei
    Fan, Bin
    Xiang, Shiming
    Pan, Chunhong
    IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2018, 29 (01) : 87 - 103
  • [4] Kernelized Locality-Sensitive Hashing
    Kulis, Brian
    Grauman, Kristen
    IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2012, 34 (06) : 1092 - 1104
  • [5] Correlated Locality-Sensitive Hashing
    Pagh, Rasmus
    ALGORITHMS - ESA 2015, 2015, 9294
  • [6] DB-LSH: Locality-Sensitive Hashing with Query-based Dynamic Bucketing
    Tian, Yao
    Zhao, Xi
    Thou, Xiaofang
    2022 IEEE 38TH INTERNATIONAL CONFERENCE ON DATA ENGINEERING (ICDE 2022), 2022, : 2250 - 2262
  • [7] Locality-Sensitive Hashing for Long Context Neural Machine Translation
    Petrick, Frithjof
    Rosendahl, Jan
    Herold, Christian
    Ney, Hermann
    PROCEEDINGS OF THE 19TH INTERNATIONAL CONFERENCE ON SPOKEN LANGUAGE TRANSLATION (IWSLT 2022), 2022, : 32 - 42
  • [8] An Improved Algorithm for Locality-Sensitive Hashing
    Cen, Wei
    Miao, Kehua
    10TH INTERNATIONAL CONFERENCE ON COMPUTER SCIENCE & EDUCATION (ICCSE 2015), 2015, : 61 - 64
  • [9] Bit Reduction for Locality-Sensitive Hashing
    Liu, Huawen
    Zhou, Wenhua
    Zhang, Hong
    Li, Gang
    Zhang, Shichao
    Li, Xuelong
    IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2024, 35 (09) : 12470 - 12481
  • [10] Cross-media retrieval based on locality-sensitive hashing and neural network algorithms
    Bai L.
    Jia Y.
    Wang H.
    Xie Y.
    Yu T.
    2018, National University of Defense Technology (40): : 93 - 98