Reinforcement learning acceleration through autonomous subgoal discovery

This paper presents two methods by which a reinforcement learning agent can automatically discover certain types of subgoals online and construct hierarchical state and action spaces. By creating useful subgoals while learning, the agent is able to accelerate learning on the current task and to transfer its expertise to other, related tasks through the reuse of its ability to attain subgoals. The presented mechanism then constructs macros action to the discovered subgoals and partitions the state space to accelerate learning time while insuring the achievablility of tasks. Simulations of different state spaces show that the policies in both original MDP and this representation achieve similar results, however the total learning time in the partition space is much smaller than the total amount of time spent on learning in the original state space.