Spatiotemporal squeeze-and-excitation residual multiplier network for video action recognition

Aiming at the shortcomings of shallow networks and general deep models in two-stream network structure, which could not effectively learn spatial and temporal information, a squeeze-and-excitation residual network was proposed for action recognition with a spatial stream and a temporal stream. Meanwhile, the long-term temporal dependence was captured by injecting the identity mapping kernel into the network as a temporal filter. Spatiotemporal feature multiplication fusion was used to further enhance the interaction between spatial information and temporal information of squeeze-and-excitation residual networks. Simultaneously, the influence of spatial-temporal stream multiplication fusion methods, times and locations on the performance of action recognition was studied. Given the limitations of performance achieved by a single model, three different strategies were proposed to generate multiple models, and the final recognition result was obtained by integrating these models through averaging and weighted averaging. The experimental results on the HMDB51 and UCF101 datasets show that the proposed spatiotemporal squeeze-and-excitation residual multiplier networks can effectively improve the performance of action recognition. © 2019, Editorial Board of Journal on Communications. All right reserved.