Adaptive Riemannian stochastic gradient descent and reparameterization for Gaussian mixture model fitting

Recent advances in manifold optimization for the Gaussian mixture model (GMM) have gained increasing interest. In this work, instead of directly addressing the manifold optimization on covariance matrices of GMM, we consider the GMM fitting as an optimization of the density function over a statistical manifold and seek the natural gradient to speed up the optimization process. We present an upper bound for the Kullback-Leibler (KL) divergence between two GMMs and obtain simple closed-form expressions for the natural gradients. With the natural gradients, we then apply the Riemannian stochastic gradient descent (RSGD) algorithm to optimize covariance matrices on a symmetric and positive definite (SPD) matrix manifold. We further propose a Riemannian Adam (RAdam) algorithm that extends the momentum method and adaptive learning in the Euclidean space to the SPD manifold space. Extensive simulations show that the proposed algorithms scale well to high-dimensional large-scale datasets and outperform expectation maximization (EM) algorithms in fitted log-likelihood.