Theoretical analysis and classification of training problem in neural networks

A problem of training is of immense importance in neural network field. In a recent history of the field several training techniques have been developed. Training is primarily seen as an optimization task. Particular popularity gained first order optimization techniques with linear convergence rates. Theoretical concepts of linear convergence rates of first order optimization techniques allow formulation of noncontroversial classification framework. The proposed classification framework permits independent specification of functions (or optimization tasks) and optimization techniques (or learning algorithms). Within this framework the problem of training three-layer MLP networks, given their mappings, is classified as PD(1,0) problem. The presented theoretical material furthermore allow direct design of universal superlinear first order techniques.