Cross-Modal Retrieval Algorithm for Image and Text Based on Pre-Trained Models and Encoders

At present, the mainstream image-text cross-modal retrieval model architectures are mainly include two kinds of model architectures based on dual encoders andfusion encoders. Although the architecture based on dual encoders has high retrieval efficiency, its accuracy is insufficient. The architecture based on fusion encodershas high retrieval accuracy but low efficiency. In order to solve the problems of the above model architecture, proposes a newcross-modal image retrieval algorithm. Firstly, a recall sequencing strategy is proposed, which uses dual encoder to achieve rough recall and fusion encoder to achieve precise sequencing. Secondly, a method to build dual encoders and fusion encoders based on multi-channel Transformer pre-trained model is proposed to achieve high-quality semantic alignment between texts and images and improve retrieval performance. Experimental results on two public datasets microsoft common objects in context and Flickr30k demonstrate the effectiveness of the proposed algorithm. © 2023 Beijing University of Posts and Telecommunications. All rights reserved.