DUALISTIC GEOMETRY OF THE MANIFOLD OF HIGHER-ORDER NEURONS

A set of neural networks, in particular the set of all the neurons of higher-order, forms a geometrical manifold. Specifically, the sets N(k)(k = 1,2,...) of the kth order neurons constitute a hierarchy of manifolds N(n) superset-of N(n-1) superset-of ... superset-of N1, where n is the number of inputs. A natural geometry is introduced to N(k) and characteristics of higher-order neurons are studied therefrom. A Riemannian metric is defined in N(k) and a dual pair of affine connections are introduced in these manifolds. A higher-order neuron realizes a transformation from vector inputs to a scalar output. Given a transformation, the approximation problem searches for its best approximation by a higher-order neuron. The best approximation is proved to be obtained by the dual geodesic projection (Projection Theorem). Moreover, the approximation error is decomposed into a sum of contributions corresponding to various orders of higher-order interactions (Decomposition Theorem). The accuracy of statistical estimation is also shown in terms of the dimensionality of a model and the number of examples (Estimation Theorem). This paper proposes an information geometrical method, which can be applied to more general neural network manifolds.