Semantic features and high-order physical features fusion for action recognition

Human action recognition (HAR) is one of the most challenging tasks in the field of computer vision due to complex backgrounds and ambiguity action, etc. To tackle these issues, we propose a novel action recognition framework called Semantic Feature and High-order Physical Feature Fusion (SF-HPFF). Concretely, we first calculate attention pooling module with a low-rank approximation to remove the information of irrelevant complex backgrounds and thus capture the interested target motion region. On this basis, motion features based on the physical characteristics of flow field and semantic features based on word embedding are developed to distinguish ambiguity behaviors. These features are of low dimension and high discrimination, which help to reduce computation burden significantly while maintaining an excellent recognition performance. Finally, cascaded convolutional fusion network is adopted to fuse features and accomplish classification. Multiple experiment results validate that the proposed SF-HPFF outperforms the state-of-art action recognition methods.