This paper studies the multi-port berth allocation and time-invariant quay crane assignment problem with speed optimization (MBACAP). In the MBACAP, each vessel visits multiple ports, while the port operators allocate berths and quay cranes (QCs) to the arriving vessels. The objective of the MBACAP is to minimize the sum of vessel fuel consumption cost, waiting cost, delay cost and QC handling cost while meeting the constraints related to vessel sailing, berth allocation and QC assignment. A mixed integer liner programming (MILP) model for the MBACAP is formulated for the first time, and an improved adaptive large neighborhood search (IALNS) algorithm is developed to solve it. In the IALNS, an initial solution generation strategy is proposed, while modified removal and insertion operators combined with the MBACAP features are devised. Moreover, several speed decision operators are proposed to optimize vessel speed during insertion operations. The proposed MILP and IALNS are tested on the MBACAP instances based on real data. The numerical experimental results show that the MILP can be solved by CPLEX optimally for some small scaled instances, while the IALNS can efficiently solve instances of all scales. The importance of considering QC assignment in MBACAP and the influence of vessel fuel consumption in ports are analyzed by comparing the numerical results of different models. In addition, the impact of different fuel prices on the MBACAP is investigated through sensitivity analysis with management insights provided.