Deep Reinforcement Learning for Robotic Control with Multi-Fidelity Models

Deep reinforcement learning (DRL) can be used for the development of robotic controllers. Complicated kinematic relationships can be learned by a DRL agent, which will result in a control policy that takes actions based on an observed state. However, a DRL agent typically goes through much trial and error before beginning to take appropriate actions. Therefore, it is often useful to leverage simulated robotic manipulators before performing any training or testing on actual hardware. There are several options for such simulation, ranging from simple kinematic models to more complex models seeking to accurately simulate the effects of gravity, inertia, and friction. The latter models can provide excellent representations of a robotic plant, but typically with a noticeably increased computational expense. Reducing the expense of simulating the robotic plant (while still maintaining a reasonable degree of accuracy) can accelerate an already expensive DRL training loop. In this work, we present a methodology for using a lower-fidelity model (based on Denavit-Hartenberg parameters) to initialize the training of a DRL agent for control of a Sawyer robotic arm. We show that the trained DRL policy can then be fine-tuned in a higher-fidelity simulation provided by the robot's manufacturer. We demonstrate the accuracy of the fully trained policy by transferring it to the actual hardware, demonstrating the power of DRL to learn complicated robotic tasks entirely in simulation. Finally, we benchmark the time required to train a policy using each level of fidelity. Copyright (c) 2022 The Authors. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0)