Comparison of Gradient Descent Methods in Online Fuzzy Co-clustering

Fuzzy co-clustering schemes including Fuzzy Co-Clustering induced by Multinomial Mixture models (FCCMM) are promising approaches for analyzing object-item cooccurrence information such as document-keyword frequencies and customer-product purchase history transactions. However, such cooccurrence datasets are generally maintained as very large matrices and cannot be dealt with conventional batch algorithms. In order to deal with such problems, online FCCMM (OFCCMM) that sequentially loads a single object has been proposed. Conventional OFCCMM uses stochastic gradient descent (SGD) to update parameters. SGD generally has drawbacks that convergence is slow and it is susceptible to vibration state and a saddle point. Many improvements on SGD have been proposed such as Momentum SGD, Nesterov's accelerated gradient method, AdaGrad, and Adam. In this study, we introduce various gradient descent methods into OFCCMM and observe their characteristics and performance through numerical experiments.