An information theory based learning paradigm for linear feature extraction

This paper presents a novel unsupervised learning paradigm for feature extraction in linear networks under the constraint that no information distortion occurs in the input-output map. Independent feature extraction is then accomplished by minimizing an appropriate information theory based measure of the statistical dependence between the output components. In the case of Gaussian input distribution, a learning rule which preserves the rotation property of the input-output map is derived by using the Lyapunov function type approach. This learning rule is shown to converge to the solution of the standard PCA. Different learning rules are derived for the case of an arbitrary input distribution. They are based on the Edgeworth expansion of a probability density function as well as on the suitably chosen parameterization of admissable linear input/output maps. Examples which validate the introduced learning paradigm are presented.