Complex sequential understanding through the awareness of spatial and temporal concepts

Understanding sequential information is a fundamental task for artificial intelligence. Current neural networks attempt to learn spatial and temporal information as a whole, limiting their abilities to represent large-scale spatial representations over long-range sequences. Here, we introduce a new modelling strategy-'semi-coupled structure' (SCS)-which consists of deep neural networks that decouple the complex spatial and temporal concepts during learning. SCS can learn to implicitly separate input information into independent parts and process these parts separately. Experiments demonstrate that SCS can successfully sequentially annotate the outline of an object in images and perform video action recognition. As an example of sequence-to-sequence problems, SCS can predict future meteorological radar echo images based on observed images. Taken together, our results demonstrate that SCS has the capacity to improve the performance of long short-term memory (LSTM)-like models on large-scale sequential tasks. Current neural networks attempt to learn spatial and temporal information as a whole, limiting their ability to process complex video data. Pang et al. improve performance by introducing a network structure which learns to implicitly decouple complex spatial and temporal concepts.