Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation

被引:65
|
作者
Meng, Qingyan [1 ,2 ]
Xiao, Mingqing [3 ]
Yan, Shen [4 ]
Wang, Yisen [3 ,5 ]
Lin, Zhouchen [3 ,5 ,6 ]
Luo, Zhi-Quan [1 ,2 ]
机构
[1] Chinese Univ Hong Kong, Shenzhen, Peoples R China
[2] Shenzhen Res Inst Big Data, Shenzhen, Peoples R China
[3] Peking Univ, Sch Artificial Intelligence, Key Lab Machine Percept MoE, Beijing, Peoples R China
[4] Peking Univ, Ctr Data Sci, Beijing, Peoples R China
[5] Peking Univ, Inst Artificial Intelligence, Beijing, Peoples R China
[6] Peng Cheng Lab, Shenzhen, Guangdong, Peoples R China
基金
中国国家自然科学基金;
关键词
D O I
10.1109/CVPR52688.2022.01212
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. However, it is a challenge to efficiently train SNNs due to their non-differentiability. Most existing methods either suffer from high latency (i.e., long simulation time steps), or cannot achieve as high performance as Artificial Neural Networks (ANNs). In this paper, we propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance that is competitive to ANNs yet with low latency. First, we encode the spike trains into spike representation using (weighted) firing rate coding. Based on the spike representation, we systematically derive that the spiking dynamics with common neural models can be represented as some sub-differentiable mapping. With this viewpoint, our proposed DSR method trains SNNs through gradients of the mapping and avoids the common non-differentiability problem in SNN training. Then we analyze the error when representing the specific mapping with the forward computation of the SNN. To reduce such error, we propose to train the spike threshold in each layer, and to introduce a new hyperparameter for the neural models. With these components, the DSR method can achieve state-of-the-art SNN performance with low latency on both static and neuromorphic datasets, including CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10.
引用
收藏
页码:12434 / 12443
页数:10
相关论文
共 50 条
  • [41] Training Deep Convolutional Spiking Neural Networks With Spike Probabilistic Global Pooling
    Lian, Shuang
    Liu, Qianhui
    Yan, Rui
    Pan, Gang
    Tang, Huajin
    NEURAL COMPUTATION, 2022, 34 (05) : 1170 - 1188
  • [42] A Low-Latency Inference of Randomly Wired Convolutional Neural Networks on an FPGA
    Kuramochi, Ryosuke
    Nakahara, Hiroki
    IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, 2021, E104D (12) : 2068 - 2077
  • [43] High-Performance Spiking Neural Network Simulator
    Khun, Jiri
    Novotny, Martin
    Skrbek, Miroslav
    2019 8TH MEDITERRANEAN CONFERENCE ON EMBEDDED COMPUTING (MECO), 2019, : 88 - 91
  • [44] High-performance deep spiking neural networks with 0.3 spikes per neuron
    Stanojevic, Ana
    Wozniak, Stanislaw
    Bellec, Guillaume
    Cherubini, Giovanni
    Pantazi, Angeliki
    Gerstner, Wulfram
    NATURE COMMUNICATIONS, 2024, 15 (01)
  • [45] High-Bandwidth Low-Latency Approximate Interconnection Networks
    Fujiki, Daichi
    Ishii, Kiyo
    Fujiwara, Ikki
    Matsutani, Hiroki
    Amano, Hideharu
    Casanova, Henri
    Koibuchi, Michihiro
    2017 23RD IEEE INTERNATIONAL SYMPOSIUM ON HIGH PERFORMANCE COMPUTER ARCHITECTURE (HPCA), 2017, : 469 - 480
  • [46] LIMITS TO LOW-LATENCY COMMUNICATION ON HIGH-SPEED NETWORKS
    THEKKATH, CA
    LEVY, HM
    ACM TRANSACTIONS ON COMPUTER SYSTEMS, 1993, 11 (02): : 179 - 203
  • [47] CS-QCFS: Bridging the performance gap in ultra-low latency spiking neural networks
    Yang, Hongchao
    Yang, Suorong
    Zhang, Lingming
    Dou, Hui
    Shen, Furao
    Zhao, Jian
    NEURAL NETWORKS, 2025, 184
  • [48] Direct Training via Backpropagation for Ultra-Low-Latency Spiking Neural Networks with Multi-Threshold
    Xu, Changqing
    Liu, Yi
    Chen, Dongdong
    Yang, Yintang
    SYMMETRY-BASEL, 2022, 14 (09):
  • [49] Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State
    Xiao, Mingqing
    Meng, Qingyan
    Zhang, Zongpeng
    Wang, Yisen
    Lin, Zhouchen
    ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 34 (NEURIPS 2021), 2021,
  • [50] Low-Latency Software Defined Network for High Performance Clouds
    Rad, Paul
    Boppana, Rajendra V.
    Lama, Palden
    Berman, Gilad
    Jamshidi, Mo
    2015 10TH SYSTEM OF SYSTEMS ENGINEERING CONFERENCE (SOSE), 2015, : 486 - 491