Data Augmentation in Latent Space with Variational Autoencoder and Pretrained Image Model for Visual Reinforcement Learning

In this paper we investigate alternative data augmentation strategies for Visual Reinforcement Learning and explore the potential benefits of fine-tuning a pretrained image encoder to enhance the learning process. We propose an innovative approach that applies data augmentation in the latent space, rather than directly manipulating pixel values. This method utilizes a Variational Autoen- coder, integrated with a pretrained image model, to facilitate the data augmentation process in a more abstract and feature-rich latent space. We use the DeepMind Control suite as a benchmark to evaluate the impact of our approach.