Adaptive Model Learning method for Reinforcement Learning

The original Q-learning method is difficult on achieving sample efficiency such as training a policy to get to a goal with in limited time step. So, the Dyna-Q agent is proposed to speed up the policy learning. However, the Dyna-Q did not specify how to build the model, so the table is used to be the model largely. In this paper, we proposed an adaptive model learning method based on tree structures and combined with Q-Learning to form Tree-Based Dyna-Q agent to enhance the policy learning. When the tree-based model learns an accurate model, a planning method can use the model to produce simulated experiences to accelerate value iterations. Thus, the agent with the proposed method can obtain virtual experiences for updating the policy. The simulation result shows that training time of our method can improve obviously.