The orderly charge and discharge scheduling of Electric Vehicles (EVs) plays an increasingly important role in safe operation of the distribution network. Currently, there exists the problem that some charging station agents do not pay attention to the difference between the charging information of EVs and the charging demand of users, and treat the charge and discharge scheduling of EVs undifferentiated, which results in users' dissatisfaction. This paper presents a three-stage optimization of charging scheduling of EVs, which is based on the electricity price to guide charging selection of users and the continuous charging and discharging of EVs while scheduling. In first stage, the agents provide different charging modes for users to choose from, a charging selection model for users based on EV charging information and the time-of-use (TOU) electricity price is established. The arrival and departure time of EVs, state-of-charge (SOC) and TOU electricity price are all factors that affect users' charging selection. Differential group dispatching of EVs with different charging modes to realize the requirements of the power grid and users. In second stage, based on the charging selection of users and the charging information of EVs, the charging load in different periods can be predicted, which can guide the power purchase for charging agents. The third stage is the integrated charging scheduling of EVs. EVs dispatching is based on the priority, which takes the charge-discharge switching times of EVs into account. The model takes the maximum income of the charging agents as the objective and takes the requirements of the power grid and users as the constraints. Modified adaptive step size fruit fly optimization algorithm (MASSFOA) is utilized to optimize the TOU electricity price, and the optimized TOU electricity price is used to guide EV user's charging selection. The results of simulation analysis show that the proposed strategy can reduce the peak-to-valley difference of the load curve, improve the continuous charging and discharging of EVs, and increase the income of charging station agents.