MULTILAYERED NEURAL NETWORK WITH AN AMSGrad OPTIMIZATION LEARNING METHOD

Abstract

In this research, we have tested the AMSGrad optimization learning method for a multilayered neural network using the logistic function. This function describes the doubling process of the local minima number and the Fourier spectra for the error function. As a result of neural network retraining, the learning error function on each neuron is characterized by a set of wave vectors of different periodicities. The average value of the learning error for all neurons can be considered an average value for all existing periodicities. At the same time, the wave vector of the total oscillation can take both commensurate and incommensurate values. The appearance of local minima is shown to be caused by the non-homogeneous learning of the neural network, which is related to the retraining of individual neurons. As the local minimum number increases with the learning rate, so does the number of such neurons.

The AMSGrad optimization method reduces the number of retrained neurons by controlling the exponential rate of average gradients decay and the square of the error objective function gradient. In other words, the learning rate of each neuron is corrected, which removes this system's degeneracy by preventing the processes of each neuron's retraining.

Keywords: multilayered neural network, AMSGrad method, local minimums, block structure.