The Impact of Synchronization in Parallel Stochastic Gradient Descent

被引：0
|
作者：
Backstrom, Karl [1 ]
Papatriantafilou, Marina [1 ]
Tsigas, Philippas [1 ]
机构：
[1] Chalmers Univ Technol, Dept Comp Sci & Engn, Gothenburg, Sweden
来源：
DISTRIBUTED COMPUTING AND INTELLIGENT TECHNOLOGY, ICDCIT 2022 | 2022年 / 13145卷
基金：
瑞典研究理事会;
关键词：
Stochastic gradient descent; Lock-free; Machine Learning;
D O I：
10.1007/978-3-030-94876-4_4
中图分类号：
TP31 [计算机软件];
学科分类号：
081202 ; 0835 ;
摘要：
In this paper, we discuss our and related work in the domain of efficient parallel optimization, using Stochastic Gradient Descent, for fast and stable convergence in prominent machine learning applications. We outline the results in the context of aspects and challenges regarding synchronization, consistency, staleness and parallel-aware adaptiveness, focusing on the impact on the overall convergence.
引用
收藏
页码：60 / 75
页数：16
相关论文
共 50 条
  • [21] A parallel and distributed stochastic gradient descent implementation using commodity clusters
    Robert K. L. Kennedy
    Taghi M. Khoshgoftaar
    Flavio Villanustre
    Timothy Humphrey
    Journal of Big Data, 6
  • [22] A Stochastic Parallel Gradient Descent Algorithm for Person Re-identification
    Cheng, Keyang
    Tao, Fei
    2018 IEEE INTERNATIONAL CONFERENCE ON VISUAL COMMUNICATIONS AND IMAGE PROCESSING (IEEE VCIP), 2018,
  • [23] EventGraD: Event-Triggered Communication in Parallel Stochastic Gradient Descent
    Ghosh, Soumyadip
    Gupta, Vijay
    2020 IEEE/ACM WORKSHOP ON MACHINE LEARNING IN HIGH PERFORMANCE COMPUTING ENVIRONMENTS (MLHPC 2020) AND WORKSHOP ON ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING FOR SCIENTIFIC APPLICATIONS (AI4S 2020), 2020, : 1 - 8
  • [24] A parallel and distributed stochastic gradient descent implementation using commodity clusters
    Kennedy, Robert K. L.
    Khoshgoftaar, Taghi M.
    Villanustre, Flavio
    Humphrey, Timothy
    JOURNAL OF BIG DATA, 2019, 6 (01)
  • [25] Adaptive wavefront control with asynchronous stochastic parallel gradient descent clusters
    Vorontsov, Mikhail A.
    Carhart, Gary W.
    JOURNAL OF THE OPTICAL SOCIETY OF AMERICA A-OPTICS IMAGE SCIENCE AND VISION, 2006, 23 (10) : 2613 - 2622
  • [26] Unforgeability in Stochastic Gradient Descent
    Baluta, Teodora
    Nikolic, Ivica
    Jain, Racchit
    Aggarwal, Divesh
    Saxena, Prateek
    PROCEEDINGS OF THE 2023 ACM SIGSAC CONFERENCE ON COMPUTER AND COMMUNICATIONS SECURITY, CCS 2023, 2023, : 1138 - 1152
  • [27] Preconditioned Stochastic Gradient Descent
    Li, Xi-Lin
    IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2018, 29 (05) : 1454 - 1466
  • [28] Stochastic Reweighted Gradient Descent
    El Hanchi, Ayoub
    Stephens, David A.
    Maddison, Chris J.
    INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 162, 2022,
  • [29] Stochastic gradient descent tricks
    Bottou, Léon
    Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2012, 7700 LECTURE NO : 421 - 436
  • [30] Byzantine Stochastic Gradient Descent
    Alistarh, Dan
    Allen-Zhu, Zeyuan
    Li, Jerry
    ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 31 (NIPS 2018), 2018, 31
12345