Ice-adhesion to various materials is a significant concern in all fields. In this paper, the ice-adhesion of asphalt pavement was studied based on the perspective of pavement surface characteristics. Firstly, the ice-adhesion characteristics of asphalt binder, aggregate, asphalt mixture with different surface structures and exposed areas of aggregate (EAA) were analyzed by pull-off test. Thereafter the surface characteristics of the asphalt mixture were studied by laser scanning method. Finally, the influence of freezing temperature on ice-pavement adhesion and the influence mechanism of surface structures on ice-pavement adhesion was explored, respectively. The results showed that the effect of asphalt binder on the ice-adhesion of pavement was not significant, while the effect of aggregate was obvious at −4 and −10 °C. Compared with diabase, gneiss and granite, limestone had better anti-icing performance. The SiO2 and Al2O3 in aggregate mainly promoted the adhesion of ice on aggregate, the CaO, MgO prevented the adhesion of ice on aggregate. With the estimated texture depth and surface area increased, the ice-adhesion strength of pavement decreased and the dropping rate was rapid at the high freezing temperature (−4 ~ −16 °C). With the EAA increased from 0 to 90%, the adhesion strength increased almost 100%. The influence of pavement structure on ice-adhesion is mainly caused by surface structure and failure mode. By studying the ice-pavement adhesion from the perspective of pavement surface characteristics, this paper may provide some fresh insights for ice-adhesion characteristics of asphalt pavement.