For simulating the structure of spline coupling in the low pressure rotor of a typical high bypass ratio turbofan engine, a mechanical model was developed and a test rotor system connected with a spline coupling was established. Appropriate support stiffness was selected through dynamics calculation in order to meet the first, second critical speed of the design requirements. Theoretical analysis and experimental research were conducted about the effects of tightening torque of the axial nuts on the connection stiffness of the spline coupling and on the dynamic characteristics including critical speed, vibration modes and unbalance responses of the rotor system. The results indicate that the tightening torque of the axial nuts increase the connection stiffness of the spline coupling with obvious nonlinearity. With increasing the tightening torque from 60 N·m to 120 N·m, the growth rates of the first-order and second-order critical speed of the rotor system are 1.23% and 0.51%, respectively, and the amplitudes of the corresponding unbalance responses increase to varying degree, while the corresponding vibration modes are basically unchanged. Changing axial and circumferential positions of unbalance value can result in different amplitudes and rate of changes of unbalance response. ©, 2015, Editorial Office of Journal of Propulsion Technology. All right reserved.