Third-order Smoothness Helps: Faster Stochastic Optimization Algorithms for Finding Local Minima

We propose stochastic optimization algorithms that can find local minima faster than existing algorithms for nonconvex optimization problems, by exploiting the third-order smoothness to escape non-degenerate saddle points more efficiently. More specifically, the proposed algorithm only needs ((O) over tilde(epsilon(-10)(/3)) stochastic gradient evaluations to converge to an approximate local minimum x, which satisfies parallel to del f (x)parallel to(2) <= epsilon and lambda(min) (del(2) f (x)) >= - root epsilon in unconstrained stochastic optimization, where (O) over tilde(.) hides logarithm polynomial terms and constants. This improves upon the (O) over tilde(epsilon(-7/2)) gradient complexity achieved by the state-of-the-art stochastic local minima finding algorithms by a factor of (O) over tilde(epsilon(-1/6)). Experiments on two nonconvex optimization problems demonstrate the effectiveness of our algorithm and corroborate our theory.