SPECIFICS OF THE LEARNING ERROR DEPENDENCE OF MULTILAYERED NEURAL NETWORKS FROM THE ACTIVATION FUNCTION DURING THE PROCESS OF PRINTED DIGITS IDENTIFICATION

Abstract

In this paper, we investigated the learning process during an identification of printed digits from the type of an activation function. The study of the activation function type and the number of iterations in the learning process of the neural system was carried out using the Fourier spectra analysis of the learning error function and branching diagrams. For this purpose, a program for the multilayer neural network was developed in the Python software, which involves setting the number of hidden layers as well as the number of neurons inside them and changes in the learning rate. The learning rate was considered as a constant, and its optimal value, where the best learning rate is observed, was determined. To analyze the learning rate effect on the educational process, we used a logistic function describing the frequency doubling process. It is shown that the learning error function is characterized by bifurcation processes leading to a chaotic state when η>0.8. The optimal learning rate value that determines the emergence of the doubling process of the local minima number is determined. It was found that the sigmoidal activation function (as compared to the activation functions ReLU and hyperbolic tangent) best satisfies the learning process of the three-layer neural network for recognizing digits, given an array of 4x7 zeros and ones. Compared to other activation functions, there is an insignificant change in the learning error during the transition from one digit to another. It is shown that an increase in the number of hidden layers does not lead to a sharp increase during the learning error. An increase in the learning iterations number is accompanied by the appearance of periodic dependences of the logistic function value of the learning rate, the period of which is a variable of the number of iterations and the learning rate. Using Fourier spectra of the error function from the learning rate value, it can be argued that an increase in the number of iterations leads to an increase in the number of harmonics, which eventually leads to the appearance of a chaotic state of the neural network.

Keywords: Multilayered neural network, activation function, optimal learning rate, digital identification.