Analysis of Stabilizing Value Iteration for Adaptive Optimal Control

Value iteration as an algorithm for 'learning' solutions to discrete-time optimal control problems is investigated in this paper. It is shown that if the iterations are initialized using a stabilizing initial guess, then the evolving control at each iteration will remain stabilizing. The novelty of this study is in providing rigorous theoretical analyses on a) continuity of the value function subject to approximation, b) stability of the system operated using any single/constant resulting control policy, c) stability of the system operated using evolving/time-varying control policy, d) convergence of the algorithm, and e) optimality of the limit function. Moreover, estimations of the region of attraction for the solution are provided so that if the initial state is within the region, the whole trajectory will remain inside it and hence, the tuned controller will remain valid for use.