MEMORY REDUCTION METHOD FOR DEEP NEURAL NETWORK TRAINING

被引：0

作者：

Shirahata, Koichi ^{[1
]}

Tomita, Yasumoto ^{[1
]}

Ike, Atsushi ^{[1
]}

机构：

[1] Fujitsu Labs Ltd, Nakahara Ku, 4-1-1 Kamikodanaka, Kawasaki, Kanagawa 2118588, Japan

来源：

2016 IEEE 26TH INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING (MLSP) | 2016年

关键词：

Deep Neural Networks; Memory Management; Accelerators;

D O I：

暂无

中图分类号：

TM [电工技术]; TN [电子技术、通信技术];

学科分类号：

0808 ; 0809 ;

摘要：

Training deep neural networks requires a large amount of memory, making very deep neural networks difficult to fit on accelerator memories. In order to overcome this limitation, we present a method to reduce the amount of memory for training a deep neural network. The method enables to suppress memory increase during the backward pass, by reusing the memory regions allocated for the forward pass. Experimental results exhibit our method reduced the occupied memory size in training by 44.7% on VGGNet with no accuracy affection. Our method also enabled training speedup by increasing the mini batch size up to double.

引用

页数：6

共 50 条

[31] A Deep Neural Network-Based Method for Building a Professional Farmer Training Model
Jing, Qiaosong
JOURNAL OF CIRCUITS SYSTEMS AND COMPUTERS, 2022, 31 (14)
[32] A memory optimal BFGS neural network training algorithm
McLoone, SF
Asirvadam, VS
Irwin, GW
PROCEEDING OF THE 2002 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS, VOLS 1-3, 2002, : 513 - 518
[33] Optimization of memory access for the convolutional neural network training
Wang J.
Hao Z.
Li H.
Xi'an Dianzi Keji Daxue Xuebao/Journal of Xidian University, 2020, 47 (02): : 98 - 107
[34] Deep memory and prediction neural network for video prediction
Liu, Zhipeng
Chai, Xiujuan
Chen, Xilin
NEUROCOMPUTING, 2019, 331 : 235 - 241
[35] Hierarchical Approximate Memory for Deep Neural Network Applications
Ha, Minho
Hwang, Seokha
Kim, Jeonghun
Lee, Youngjoo
Lee, Sunggu
2020 54TH ASILOMAR CONFERENCE ON SIGNALS, SYSTEMS, AND COMPUTERS, 2020, : 261 - 266
[36] A Survey on Memory Subsystems for Deep Neural Network Accelerators
Asad, Arghavan
Kaur, Rupinder
Mohammadi, Farah
FUTURE INTERNET, 2022, 14 (05):
[37] Towards Deep Neural Network Training on Encrypted Data
Nandakumar, Karthik
Ratha, Nalini
Pankanti, Sharath
Halevi, Shai
2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW 2019), 2019, : 40 - 48
[38] Dedicated Deep Neural Network Architectures and Methods for Their Training
Rozycki, P.
Kolbusz, J.
Wilamowski, B. M.
INES 2015 - IEEE 19TH INTERNATIONAL CONFERENCE ON INTELLIGENT ENGINEERING SYSTEMS, 2015, : 73 - 78
[39] Distributed Deep Neural Network Training on Edge Devices
Benditkis, Daniel
Keren, Aviv
Mor-Yosef, Liron
Avidor, Tomer
Shoham, Neta
Tal-Israel, Nadav
SEC'19: PROCEEDINGS OF THE 4TH ACM/IEEE SYMPOSIUM ON EDGE COMPUTING, 2019, : 304 - 306
[40] Evolving a Deep Neural Network Training Time Estimator
Pinel, Frederic
Yin, Jian-xiong
Hundt, Christian
Kieffer, Emmanuel
Varrette, Sebastien
Bouvry, Pascal
See, Simon
OPTIMIZATION AND LEARNING, 2020, 1173 : 13 - 24

← 1 2 3 4 5 →