MULTI-PARAMETER DYNAMIC LEARNING MAPS OF NEURAL NETWORKS TRAINED WITH ADAM OPTIMISER

Abstract

Background. Understanding how hyperparameters of the Adam optimizer shape the learning dynamics of multilayer neural networks remains an open problem, particularly in regimes where training transitions from stable to chaotic behavior. Previous studies have shown that learning rate and network complexity can lead to underfitting, efficient learning, overfitting, or chaotic states; however, systematic visualization of these regimes in the β₁–β₂ parameter space is still lacking.

Materials and Methods. Dynamic regime maps are constructed for a multilayer neural network trained on binary-encoded digit patterns. The training process is analyzed using a recurrent two-dimensional mapping based on the error evolution of a neuron and its neighbor. Periodicities in the final iterations are identified via the convergence of periodicities method, enabling classification of learning regimes from fixed points to high-order cycles and chaos. The maps are generated for varying learning rates α, training duration, network depth, and neuron count, with β₁ and β₂ scanned over the interval [0, 0.999].

Results and Discussion. The results demonstrate strong sensitivity of Adam dynamics to the joint values of β₁ and β₂. Small learning rates (α = 0.001) produce predominantly first-order periodicity, indicating underfitting. Moderate values (α = 0.1 - 0.4 ) reveal structured transitions from rapid to slower learning as β₂ increases. In contrast, large α (≥ 0.6) leads to fragmented, non-monotonic patterns associated with overfitting and chaotic dynamics. Increasing network size introduces noisy gradients, shrinking stable regions and expanding chaotic zones, whereas smaller networks yield smoother regime transitions.

Conclusion. Dynamic regime mapping in the β₁–β₂ space serves as an effective diagnostic tool for identifying stable, inefficient, and chaotic learning regimes. The findings confirm the high sensitivity of Adam to hyperparameter interactions and network architecture, showing that inappropriate combinations can induce chaotic behavior even at moderate learning rates. The proposed framework provides a novel approach for analyzing optimizer dynamics and guiding hyperparameter selection in deep learning.

Keywords: Adam optimizer; learning dynamics; periodicity maps; hyperparameter tuning; chaotic learning; multilayer neural networks.