Delving into CLIP latent space for Video Anomaly Recognition

被引:1
|
作者
Zanella, Luca [1 ]
Liberatori, Benedetta [1 ]
Menapace, Willi [1 ]
Poiesi, Fabio [2 ]
Wang, Yiming [2 ]
Ricci, Elisa [1 ,2 ]
机构
[1] Univ Trento, Trento, Italy
[2] Fdn Bruno Kessler, Trento, Italy
关键词
Video anomaly detection and recognition; Multi-modal learning;
D O I
10.1016/j.cviu.2024.104163
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
We tackle the complex problem of detecting and recognising anomalies in surveillance videos at the frame level, utilising only video-level supervision. We introduce the novel method AnomalyCLIP, , the first to combine Vision and Language Models (VLMs), such as CLIP, with multiple instance learning for joint video anomaly detection and classification. Our approach specifically involves manipulating the latent CLIP feature space to identify the normal event subspace, which in turn allows us to effectively learn text-driven directions for abnormal events. When anomalous frames are projected onto these directions, they exhibit a large feature magnitude if they belong to a particular class. We also leverage a computationally efficient Transformer architecture to model short- and long-term temporal dependencies between frames, ultimately producing the final anomaly score and class prediction probabilities. We compare AnomalyCLIP against state-of-the-art methods considering three major anomaly detection benchmarks, i.e. ShanghaiTech, UCF-Crime, and XD- Violence, and empirically show that it outperforms baselines in recognising video anomalies. Project website and code are available at https://lucazanella.github.io/AnomalyCLIP/.
引用
收藏
页数:13
相关论文
empty
未找到相关数据