A novel approach to reduce vibration was put forward by applying rubber concrete as backfill layer of the high-speed railway tunnel, and its feasibility was analyzed based on the vibration isolation theory. A three-dimensional spatial coupling model of vehicle-track-tunnel-rock mass was established by means of vehicle-track coupling dynamics theory. The dynamic response of the vehicle, track, and tunnel structure under common and rubber concrete backfill layer was compared. The vibration reduction performance and the characteristics of rubber concrete were discussed. The change in tunnel vibration under different elastic modulus and damping ratios of backfill layer was analyzed, and the vibration reduction effect of rubber concrete combined with damping cushion was studied. Results show that the influence of the rubber concrete backfill layer on the dynamic indices of vehicle and track structure can be neglected. Because of the application of rubber concrete, the vibration acceleration of tunnel decreases by about 40%, and 4–8 dB can be reduced in the corresponding frequency of 100–200 Hz. With decrease in the elastic modulus and increase in the damping ratio, the vibration of the tunnel decreases gradually. Moreover, the influence of damping ratio is more significant than that of elastic modulus. The combination of rubber concrete backfill layer and damping cushion demonstrates a superior effect in reducing vibration, which decreases the vibration level by 3–5 dB compared with setting the damping cushion only. The article is expected to provide theoretical guidance for the application of rubber concrete in the high-speed railway. © The Author(s) 2018.