A singular perturbation approach to fixed point learning in dynamical systems and neural networks

Supervised learning in recurrent neural networks may be formalized as the adaptation of parameters in a dynamical system for the minimization of a given optimality criterion. Gradient-based techniques for fixed point learning are pervasive and, in a dynamical system context, they can be analyzed through the singular perturbation model proposed in this paper. Time-scaling and convergence properties are furtherly characterized and some strong, hypotheses concerning the evolution of the network are relaxed. The singular perturbation framework is also valid for the formulation and analysis of different learning techniques. In particular, the incorporation of Newton methods allows gradient-based schemes to cover more general situations and reduce the dependence with respect to the choice of initial points in the search process.