SGD method for entropy error function with smoothing l0 regularization for neural networks

The entropy error function has been widely used in neural networks. Nevertheless, the network training based on this error function generally leads to a slow convergence rate, and can easily be trapped in a local minimum or even with the incorrect saturation problem in practice. In fact, there are many results based on entropy error function in neural network and its applications. However, the theory of such an algorithm and its convergence have not been fully studied so far. To tackle the issue, this works proposes a novel entropy function with smoothing l(0) regularization for feed-forward neural networks. An empirical evaluation has been conducted on real-world datasets to demonstrate that the newly conceived algorithm allows us to substantially improve the prediction performance of the considered neural networks. More importantly, the experimental results also show that the proposed function brings in more precise classifications, compared to well-founded baselines. The work is novel as it enables neural networks to learn effectively, producing more accurate predictions compared to state-of-the-art algorithms. In this respect, it is expected that the algorithm will contribute to existing studies in the field, advancing research in Machine Learning and Deep Learning.