Purpose A novel vision-based relative navigation system (VBRNS) plays an important role in aeronautics and astronautics fields, and the filter is the core of VBRNS. However, most of the existing filtering algorithms used in VBRNS are derived based on Gaussian assumption and disregard the non-Gaussianity of VBRNS. Therefore, a novel robust filtering named as cubature Huber-based filtering (CHF) is proposed and applied to VBRNS to improve the navigation accuracy in non-Gaussian noise case. Design/methodology/approach Under the Bayesian filter framework, the third-degree cubature rule is used to compute the cubature points which are propagated through state equation, and then the predicted mean and the associated covariance are taken. A combined minimum l(1) and l(2)-norm estimation method referred as Huber's criterion is used to design the measurement update. After that, the vision-based relative navigation model is presented and the CHF is used to integrate the line-of-sight measurements from vision camera with inertial measurement of the follower to estimate the precise relative position, velocity and attitude between two unmanned aerial vehicles. During the design of relative navigation filter, the quaternions are used to represent the attitude and the generalized Rodrigues parameters are used to represent the attitude error. The simulation is conducted to demonstrate the effectiveness of the algorithm. Findings By this means, the VBRNS could perform better than traditional VBRNS whose filter is designed by Gaussian filtering algorithms. And the simulation results demonstrate that the CHF could exhibit robustness when the system is non-Gaussian. Moreover, the CHF has more accurate estimation and faster rate of convergence than extended Kalman Filtering (EKF) in face of inaccurate initial conditions. Originality/value A novel robust nonlinear filtering algorithm named as CHF is proposed and applied to VBRNS based on cubature Kalman filtering (CKF) and Huber's technique. The CHF could adapt to the non-Gaussian system effectively and perform better than traditional Gaussian filtering such as EKF.