Fast principal component extraction by a weighted information criterion

Principal component analysis (PCA) is an essential technique in data compression and feature extraction, and there has been much interest in developing fast PCA algorithms. On the basis of the concepts of both weighted subspace and information maximization, this paper proposes a weighted information criterion (WINC) for searching the optimal solution of a linear neural network. We analytically show that the optimum weights globally asymptotically converge to the principal eigenvectors of a stationary vector stochastic process. We establish a dependent relation of choosing the weighting matrix on statistics of the input process through the analysis of stability of the equilibrium of the proposed criterion. Therefore, we are able to reveal the constraint on the choice of a weighting matrix. We develop two adaptive algorithms based on the WINC for extracting in parallel multiple principal components. Both algorithms are able to provide adaptive step size, which leads to a significant improvement in the learning performance. Furthermore, the recursive least squares (RLS) version of WINC algorithms has a low computational complexity O(Np), where N is the input vector dimension, and p is the number of desired principal components. In fact, the WINC algorithm corresponds to a three-layer linear neural network model capable of performing, in parallel, the extraction of multiple principal components. The WINC algorithm also generalizes some well-known PCA/PSA algorithms just by adjusting the corresponding parameters. Since the weighting matrix does not require an accurate value, it facilitates the system design of the WINC algorithm for practical applications. The accuracy and speed advantages of the WINC algorithm are verified through simulations.