Multi-step Prediction for Learning Invariant Representations in Reinforcement Learning

In this paper, we focus on how to achieve task-relevant feature representations in reinforcement learning from image observations without relying either on domain knowledge or pixel-reconstruction. Although the existing algorithms based on reconstruction and contrastive learning have achieved excellent success, the sample efficiency and robustness of the algorithm are limited due to task-irrelevant information. In this paper, we utilize bisimulation metrics to construct an invariant representation learning method and extract task-relevant information. The research shows that the multi-step prediction environment model can retain longer-term state-transition information. In addition, we propose a multi-step prediction method to collect cumulative loss and update the extractor for representing learning, thereby improving the relevance of the extracted information of the task. Experiments on tasks with or without distractors show that the proposed method achieves better results.