In view that the requirements of miniaturization, low power consumption and low cost for the navigation system, attitude measurement using magnetometer could well meet the above requirements. However, the measurement accuracy of geomagnetic field is severely affected by the external magnetic field, which would increase attitude calculating error. Therefore, it is important to do error compensation for the output of magnetometer. In this work, a novel error compensation method of magnetometer for attitude measurement using modified artificial bee colony(ABC) is proposed. After analyzing the error sources of the sensor, an equivalent error model is established. ABC is used to estimate the compensation parameters of the error model. But it's computing speed is too slow because of the initial food source is randomly generated and not reasonable. Then, a hybrid algorithm for obtaining the compensation parameters is designed, in which the recursive least square algorithm is taken to adjust the initial value of ABC Comparing with the traditional parameter estimation algorithm, the modified ABC has higher accuracy and faster efficiency. In the numerical simulation, the error model coefficients obtained by the modified algorithm are close to the theoretical coefficients. The results show that the magnetic field intensity after correction is basically consistent with the actual geomagnetic field intensity. Finally, the modified error compensation method is used to correct the output of magnetometers. As a result, the pitch angle error is reduced from +/- 6 degrees to +/- 0.8 degrees, the roll angle error is reduced from +/- 15 degrees to +/- 1 degrees by the compensation. The measurement accuracy of attitude angle is improved nearly tenfold, which can be used for conventional navigation system.