Smooth Exact Gradient Descent Learning in Spiking Neural Networks

被引：0

作者：

Klos, Christian ^{[1
]}

Memmesheimer, Raoul-Martin ^{[1
]}

机构：

[1] Univ Bonn, Inst Genet, Neural Network Dynam & Computat, D-53115 Bonn, Germany

来源：

PHYSICAL REVIEW LETTERS | 2025年 / 134卷 / 02期

关键词：

ERROR-BACKPROPAGATION; NEURONS; SIMULATION; SPARSE; CHAOS; MODEL; FIRE;

D O I：

10.1103/PhysRevLett.134.027301

中图分类号：

O4 [物理学];

学科分类号：

0702 ;

摘要：

Gradient descent prevails in artificial neural network training, but seems inept for spiking neural networks as small parameter changes can cause sudden, disruptive appearances and disappearances of spikes. Here, we demonstrate exact gradient descent based on continuously changing spiking dynamics. These are generated by neuron models whose spikes vanish and appear at the end of a trial, where it cannot influence subsequent dynamics. This also enables gradient-based spike addition and removal. We illustrate our scheme with various tasks and setups, including recurrent and deep, initially silent networks.

引用

页数：8

共 50 条

[21] Learning dynamics of gradient descent optimization in deep neural networks
Wei Wu
Xiaoyuan Jing
Wencai Du
Guoliang Chen
Science China Information Sciences, 2021, 64
[22] Learning dynamics of gradient descent optimization in deep neural networks
Wei WU
Xiaoyuan JING
Wencai DU
Guoliang CHEN
ScienceChina(InformationSciences), 2021, 64 (05) : 17 - 31
[23] INVERSION OF NEURAL NETWORKS BY GRADIENT DESCENT
KINDERMANN, J
LINDEN, A
PARALLEL COMPUTING, 1990, 14 (03) : 277 - 286
[24] ANALYSIS OF GRADIENT DESCENT LEARNING ALGORITHMS FOR MULTILAYER FEEDFORWARD NEURAL NETWORKS
GUO, H
GELFAND, SB
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS, 1991, 38 (08): : 883 - 894
[25] Dynamics of on-line gradient descent learning for multilayer neural networks
Saad, D
Solla, SA
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 8: PROCEEDINGS OF THE 1995 CONFERENCE, 1996, 8 : 302 - 308
[26] Sparse Spiking Gradient Descent
Perez-Nieves, Nicolas
Goodman, Dan F. M.
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 34 (NEURIPS 2021), 2021,
[27] Contrastive Learning in Random Neural Networks and its Relation to Gradient-Descent Learning
Romariz, Alexandre
Gelenbe, Erol
COMPUTER AND INFORMATION SCIENCES II, 2012, : 511 - 517
[28] An exact mapping from ReLU networks to spiking neural networks
Stanojevic, Ana
Wozniak, Stanislaw
Bellec, Guillaume
Cherubini, Giovanni
Pantazi, Angeliki
Gerstner, Wulfram
NEURAL NETWORKS, 2023, 168 : 74 - 88
[29] Phonetic classification with spiking neural network using a gradient descent rule
Ourdighi, A.
Lacheheb, S. E.
Benyettou, A.
SECOND INTERNATIONAL CONFERENCE ON COMPUTER AND ELECTRICAL ENGINEERING, VOL 2, PROCEEDINGS, 2009, : 36 - 40
[30] Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
Li, Yuanzhi
Liang, Yingyu
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 31 (NIPS 2018), 2018, 31

← 1 2 3 4 5 →