An improved adaptive momentum gradient descent algorithm

To improve the poor global convergence of Adam algorithm，an AngleAdam algorithm with angular coefficients was proposed． The algorithm used the angle information between two continuous gradients to adaptively control the step size，which improved the problem of poor global convergence of Adam algorithm to a certain extent，and improved the optimization ability．By using the online learning framework，the convergence of the algorithm was analyzed from the perspective of regret bound，proving that the AngleAdam had sublinear regret．Based on the constructed three non-convex functions and the depth neural network model，the optimization ability of the AngleAdam algorithm was tested． Experimental results show that the algorithm can obtain better optimization results. © 2023 Huazhong University of Science and Technology. All rights reserved.