On-line learning from finite training sets

We analyse on-line (gradient descent) learning of a rule from a finite set of training examples at non-infinitesimal learning rates eta, calculating exactly the time-dependent generalization error for a simple model scenario. In the thermodynamic limit, we close the dynamical equation for the generating function of an infinite hierarchy of order parameters using ''within-sample self-averaging''. The resulting dynamics is non-perturbative in eta, with a slow mode appearing only above a finite threshold eta(min). Optimal settings of eta for given final learning time are determined and the results are compared with offline gradient descent.