Contrastive Language-Video Learning Model Based on Spatio-Temporal Information Auxiliary Supervision

Video action recognition is one of the hot topics in the field of computer vision, which has attracted the attention of many researchers in recent decades. The basic method of video action recognition is widely used in Internet video audit, video surveillance, human-computer interaction and other fields. The main body of video is usually human. Because of the complexity and variability of human action categories and environment in real life, and the huge amount of video, it requires high computing devices, which brings great challenges to the task of video action recognition. In the field of video surveillance, most of the existing systems only record abnormal actions and cannot recognize it in real time, so they cannot realize real intelligence;while in the field of Internet video audit, a lot of manual audits is often needed, which can’t recognize human action in real time. Video can usually be regarded as images that change with time. This special image data contains rich information. To recognize actions from video, it is not only necessary to obtain the spatial information of the image at each moment, but also to capture the temporal reasoning information between frames, and more importantly, to obtain the spatio-temporal information. To this end, researchers have developed many network architectures for video action recognition tasks, which can be divided into the following four categories: two-stream convolutional neural networks (CNNs) based methods, 3D CNNs based methods, 2D convolutional network with spatio-temporal modeling module, and visual Transformer-based networks. The use of Transformer-based network models that integrate both language and image modalities has made great progress in the field of computer vision. There are three representative research works in computer vision tasks related to images:namely Contrastive Language-Image Pre-training (CLIP) model, A Large-scale Image and Noisy-text embedding (ALIGN) model and Florence model. However, when these models are applied to video recognition tasks, there are still some limitations that need to be addressed, such as the lack of consideration of rich spatiotemporal information in videos and the simplicity of text descriptions used to describe video categories, which results in insufficient contextual description ability. In this paper, we propose a language-video contrastive learning model based on spatio-temporal auxiliary information supervision. For video encoder, we propose a category token-based temporal weighted displacement module for temporal modeling, which enables temporal information to be propagated at various levels of the network from the bottom to the top. Furthermore, we propose a spatiotemporal information auxiliary supervision module to deeply explore the rich spatio-temporal information embedded in visual tokens. For language encoder, we propose a prompt learning method based on large-scale language pre-training models to extend action category text descriptions and generate text descriptions with rich contextual semantic information. The experiment has achieved better results than the current most advanced methods on four video action recognition datasets, namely, mini-Kinetics-200, Kinetics-400, UCF101, and HMDB51, and it is better than or comparable to the current state-of-the-art method, and the accuracy is 2.5%, 0.3%, 0.6% and 2.4% higher than the baseline, respectively. © 2024 Science Press. All rights reserved.