Personalized Federated Learning via Deviation Tracking Representation Learning

Federated learning preserves privacy by decentralized training of individual client devices, ensuring only model weights are shared centrally. However, the data heterogeneity across clients presents challenges. This paper focuses on representation learning, a variant of personalized federated learning. According to various studies, the representation learning model can be divided into two: the base layer, shared and updated to the server, and the head layer, localized to individual clients. The novel approach exclusively utilizes the base layer for both local and global training, arguing that the head layer might introduce noise due to data heterogeneity. This can potentially affect accuracy, and the head layer is used only for fine-tuning after training to capture unique client data characteristics. Here, we observed that prolonged base training can diminish accuracy in the post-fine-tuning. As a countermeasure, we proposed a method to determine the best round for fine-tuning based on monitoring the standard deviation of test accuracy across clients. This strategy aims to generalize the global model for all the clients before fine-tuning. The study highlights the downside of excessive base training on fine-tuning accuracy and introduces a novel approach to pinpoint optimal fine-tuning moments, thereby minimizing computational and communication overheads. Similarly, we achieved a better accuracy of 53.6% than other approaches while there's a trade-off of minute communication round.