Collaborative filtering is one of the most commonly used methods in recommendation systems. However, the sparsity of the rating matrix, cold start-up, and most recommendation algorithms only consider the users while neglecting the relationship between the products, all of what limit the effectiveness of the recommendation algorithms. In this paper, based on the self-attention mechanism, a deep learning model, named HARSAM, is proposed for modeling user interaction data and learning the user's latent preference expression. HARSAM partitions the user's latent feedback data in different time granularity and employs the self-attention mechanism to extract the correlation among the data in each partition. Moreover, the model learns the user's latent preferences through the deep neural network. Simultaneously, the model learns the item latent representation by making use of the stacked denoising autoencoder to model the item's rating data. As the result, the model recommends items to users according to the similarities between user's preference and items. Experiments conducted on the public data demonstrate the effectiveness of the proposed model.