AMU-Net: an accurate motion perception and aggregation model for action recognition

Motion information plays a vital role in video action recognition, serving as the fundamental building basis for the accurate interpretation of dynamic sequences. However, extracting accurate motion details remains a significant challenge for two-dimensional (2D) CNNs. To address this issue, we present an action recognition framework, named accurate motion understanding network (AMU-Net), designed to effectively perceive and aggregate valuable motion cues. Specifically, AMU-Net is a 2D CNN equipped with the proposed accurate motion perceptron (AMP) and action graph module (AGM). To capture finer local motion details, the AMP is introduced to handle motion noise in temporal differences. This module enables the extraction of critical local motion patterns from bidirectional temporal differences and enhances action-related features. Furthermore, to learn more precise global motion representations, the AGM is introduced to address the spatial sparsity of motion objects by detecting motion objects and selectively aggregating their features using a graph reasoning framework. Extensive experiments are conducted on three public benchmarks: ActivityNet-200, UCF-101, and Kinetics-400. Experimental results demonstrate that the proposed AMU-Net (based on ResNet-50) outperforms recent 2D CNN-based methods with a comparable computational overhead. In addition, the experimental results also show the effective transferability of the two modules to three popular lightweight convolutional architectures, emphasizing their versatility. (c) 2024 SPIE and IS&T