With epidemic outbreak, individuals are afraid of being infected, become cautious and adopt behavioral changes to reduce their probability of being infected especially at high infective level. This phenomenon is regarded as the psychological effect in the population, which occurs not only in the susceptible population but also in the vaccinated population. In this paper, considering the psychological effects of two populations, a new SIVS model with nonmonotone incidence rate and imperfect vaccination is constructed on the scale-free network, which is more closely related to the actual spread of epidemics. Based on the model, existence conditions of multiple endemic equilibrium points and two threshold parameters are firstly derived. Next, a necessary and sufficient condition which determines the occurrence of a backward bifurcation at R0 = 1 is obtained. Besides, the global asymptotical stability of disease-free equilibrium and the persistence of the disease are proved. By using the monotone iterative technique, the global attractivity of the unique endemic equilibrium is analyzed. And the optimal vaccinated strategy is studied by the method of Pontryagin's maximum principle. Finally, through numerical simulations, the interaction and impact of the psychological effects, vaccines, and disease outbreaks are revealed.