Stochastic Markov gradient descent and training low-bit neural networks

被引：0

作者：

Ashbrock, Jonathan ^{[1
]}

Powell, Alexander M. ^{[2
]}

机构：

[1] MITRE Corp, Mclean, VA 22102 USA

[2] Vanderbilt Univ, Dept Math, Nashville, TN 37240 USA

来源：

SAMPLING THEORY SIGNAL PROCESSING AND DATA ANALYSIS | 2021年 / 19卷 / 02期

关键词：

Neural networks; Quantization; Stochastic gradient descent; Stochastic Markov gradient descent; Low-memory training;

D O I：

10.1007/s43670-021-00015-1

中图分类号：

O29 [应用数学];

学科分类号：

070104 ;

摘要：

The massive size of modern neural networks has motivated substantial recent interest in neural network quantization, especially low-bit quantization. We introduce Stochastic Markov Gradient Descent (SMGD), a discrete optimization method applicable to training quantized neural networks. The SMGD algorithm is designed for settings where memory is highly constrained during training. We provide theoretical guarantees of algorithm performance as well as encouraging numerical results.

引用

页数：23

共 50 条

[41] Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks
Zhou, Aojun
Yao, Anbang
Wang, Kuan
Chen, Yurong
2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2018, : 9426 - 9435
[42] ASYNCHRONOUS STOCHASTIC GRADIENT DESCENT FOR DNN TRAINING
Zhang, Shanshan
Zhang, Ce
You, Zhao
Zheng, Rong
Xu, Bo
2013 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2013, : 6660 - 6663
[43] Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
Li, Yuanzhi
Liang, Yingyu
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 31 (NIPS 2018), 2018, 31
[44] Simple Evolutionary Optimization Can Rival Stochastic Gradient Descent in Neural Networks
Morse, Gregory
Stanley, Kenneth O.
GECCO'16: PROCEEDINGS OF THE 2016 GENETIC AND EVOLUTIONARY COMPUTATION CONFERENCE, 2016, : 477 - 484
[45] Crossprop: Learning Representations by Stochastic Meta-Gradient Descent in Neural Networks
Veeriah, Vivek
Zhang, Shangtong
Sutton, Richard S.
MACHINE LEARNING AND KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2017, PT I, 2017, 10534 : 445 - 459
[46] Is Learning in Biological Neural Networks Based on Stochastic Gradient Descent? An Analysis Using Stochastic Processes
Christensen, Soeren
Kallsen, Jan
NEURAL COMPUTATION, 2024, 36 (07) : 1424 - 1432
[47] Stability analysis of stochastic gradient descent for homogeneous neural networks and linear classifiers
Paquin, Alexandre Lemire
Chaib-draa, Brahim
Giguere, Philippe
NEURAL NETWORKS, 2023, 164 : 382 - 394
[48] INVERSION OF NEURAL NETWORKS BY GRADIENT DESCENT
KINDERMANN, J
LINDEN, A
PARALLEL COMPUTING, 1990, 14 (03) : 277 - 286
[49] Gradient Descent for Spiking Neural Networks
Huh, Dongsung
Sejnowski, Terrence J.
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 31 (NIPS 2018), 2018, 31
[50] Performances and Trade-offs of Low-Bit Precision Neural Networks based on Resistive Memories
Zanotti, Tommaso
Pavan, Paolo
Puglisi, Francesco Maria
2021 IEEE INTERNATIONAL INTEGRATED RELIABILITY WORKSHOP (IIRW), 2021, : 7 - 11

← 1 2 3 4 5 →