Federated learning (FL) has been widely adopted for intelligent skin lesion diagnosis due to enabling collaborative model training across distributed data sources while preserving data privacy. However, traditional centralized learning paradigm, which collects data from distributed institutions to train neural models centrally, has the risk of privacy leakage. FL enables collaborative learning on training an artificial intelligence (AI) model among scattered medical institutions (MIs) without directly sharing raw skin lesion data, which is a promising solution. However, FL generally suffers from performance deterioration due to heterogeneous skin lesion data sets, and the differences in computing power and communication conditions among different MIs can impact the training efficiency. In this article, we present a prototypes contrastive learning (PCL) empowered intelligent diagnosis mechanism for skin lesion. Our mechanism designs a PCL-based local training to overcome data heterogeneity by designing two special losses, where prototypes-based contrastive loss can increase the interclass variance and reduce the intraclass variance of various skin lesions, and prototypes-based consistent regularization loss can prevent the shift of local updating. Meanwhile, a hierarchical FL aggregation mechanism is proposed, where asynchronous and synchronous aggregations are combined to ensure the training efficiency in case of large differences in computing power and communication conditions. Finally, simulation results show that our developed method can effectively mitigate the adverse impact of heterogeneity skin lesion data sets and provide efficient FL training.