Adaptive multilayer perceptrons with long- and short-term memories

Multilayer perceptrons (MLPs) with long- and short-term memories (LASTMs) are proposed for adaptive processing. The activation functions of the output neurons of such a network are linear, and thus the weights in the last layer affect the outputs of the network linearly and are called linear weights. These linear weights constitute the short-term memory and other weights the long-term memory. It is proven that virtually any function f(x, theta) with an environmental parameter theta can be approximated to any accuracy by an MLP with LASTMs whose long-term memory is independent of theta. This independency of theta allows the long-term memory to be determined in an a priori training and allows the on-line adjustment of only the short-term memory for adapting to the environmental parameter theta. The benefits of using an MLP with LASTMs include less on-line computation, no poor local extrema to fall into, and much more timely and better adaptation. Numerical examples illustrate that these benefits are realized satisfactorily.