Three-dimensional spatial coordinate measurement system including two optical theodolites is a kind of common extrinsic calibration equipment. During the course of geometrical dimensional measurement, there will be system error of vertical observing angle's reading in theodolite's certain position, the results of which leads to bad effects of precision. Grounded on actual optical theodolites measurement system, objected to two dimension coordinate simulator, this paper has built the mathematical model of coordinate dot to be measured and observing angle of theodolite taking system error as nonlinear component. The Neural Network (NN) to simulate the nonlinear relationship of observing angles and distances has been established taking the data satisfied by the conditions of error existence for training samples. The idea is provided of acquiring nonlinear mapping relations by NN approaching to compensate system error. In the experiment, the weights of NN have been continuously optimized to get the optimal parameter configuration, and consistence degree of the compensated results and theoretical results has validated this methods.