Mean squared error analysis of analog neural networks subject to drifting targets and noise

In previous work we studied the tracking behaivor of neural networks with binary outputs subject 20 drifting targets and noise. This paper extends this work by considering the tracking behaivor of analog output neutrons when subjected to additive noise and slowly drifting target weights. The target weights are described by a stochastic difference equation with weights changing slowly with time. The tracker weights follow the Least ii-lean Square (LMS) gradient descent algorithm and at each update are given a noise corrupted value of the output of the target network. When inputs are Gaussian and the activation used is the Gaussian error function (closely approximates the standard sigmoidal activation function) the analysis is tractable. The dynamics of target and tracking networks are described by a set of stochastic difference equations. We obtain an approximation of the mean squared generalization error by linearizing the nonlinear difference equations and using simple probabilistic arguments. We consider the single neuron case and same specific multi-layer neural networks.