PHASED INTEGRATION OF NEURAL NETWORKS OF DIFFERENT ARCHITECTURES IN MATHEMATICAL COMPUTING SYSTEMS

Mykhailo Bavdys, Oleksii Kushnir

Abstract


Background. The development of modern mathematical computing systems requires the effective implementation of machine learning algorithms while maintaining a balance between prediction accuracy and computational resources. Particular attention should be given to the phased integration of neural networks of varying complexity with minimized risks for production systems and investigation of the saturation effect when increasing architectural depth.

Materials and Methods. This article aims to develop a methodology for evolutionary integration of neural networks from simple perceptrons to ultra-deep architectures in mathematical computing systems, with detailed comparative analysis of four architectural types and mathematical modeling of the accuracy saturation effect.

Results and Discussion. For this purpose, four neural network architectures were investigated: a single-layer perceptron, a four-layer network (128→64→32), a ten-layer network (128→96→64→48→32→24→16→12), and a twenty-layer architecture with gradual dimensionality reduction. Experiments were conducted on a dataset from mathematical modeling results containing 45,000 samples with 24 characteristics. A comprehensive system of metrics was used to evaluate accuracy, processing speed, resource consumption, and model stability. The experimental design included stratified data splitting and cross-validation to ensure statistical reliability of the obtained results across different architectural configurations.

Conclusion. As a result, the single-layer perceptron demonstrated baseline accuracy of 78.3% with minimal resource consumption (45 MB RAM, 15 ms latency). The four-layer network achieved 94.1% accuracy with a moderate increase in resource costs. The ten-layer architecture showed 95.6% accuracy, demonstrating the beginning of the saturation effect. The twenty-layer network achieved only 96.8% accuracy with disproportionate growth in resource consumption (1024 MB RAM, 270 ms latency). Mathematical modeling confirmed the logistic nature of the relationship between accuracy and architectural complexity. The findings provide practical guidelines for selecting optimal neural network architectures in resource-constrained production environments, establishing clear thresholds beyond which increased complexity yields diminishing returns.

Keywords: Neural networks, evolutionary integration, mathematical computing, deep learning, saturation effect, architecture optimization


Full Text:

PDF

References


[1] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444. https://doi.org/10.1038/nature14539

[2] Stipsitz, M., Sanchis-Alepuz, H. Approximating the steady-state temperature of 3D electronic systems with convolutional neural networks. Mathematical and Computational Applications, 2022, 27(1), 7. https://doi.org/10.3390/mca27010007

[3] Kang, S., Jung, J., Lee, S. A hybrid deep learning model predicting pressure distribution images for 2D airfoils from coordinate information. Nature Scientific Reports, 2024, 14, 30764. https://doi.org/10.1038/s41598-024-84940-w

[4] Bolesta, I., Kushnir, O., Bavdys, M., Khvyshchun, I., Demchuk, A. Computational-Measurement System "Nanoplasmonics". Part 1: Architecture. 2019 XIth International Scientific and Practical Conference on Electronics and Information Technologies (ELIT). IEEE, 2019, pp. 164-167. https://doi.org/10.1109/ELIT.2019.8892288

[5] Bolesta, I., Kushnir, O., Bavdys, M., Khvyshchun, I., Demchuk, A. Computational-Measurement System "Nanoplasmonics". Part 2: Structure of Microservices. 2019 XIth International Scientific and Practical Conference on Electronics and Information Technologies (ELIT). IEEE, 2019, pp. 172-175. https://doi.org/10.1109/ELIT.2019.8892345

[6] Li, N., Ma, L., Xing, T., Yu, G., Wang, C., Wen, Y., Cheng, S., Gao, S. Automatic design of machine learning via evolutionary computation: A survey. Applied Soft Computing, 2023, 143, 110412. https://doi.org/10.1016/j.asoc.2023.110412

[7] Ying, H., Song, M., Tang, Y., Xiao, S., Xiao, Z. Enhancing deep neural network training efficiency and performance through linear prediction. Scientific Reports, 2024, 14(1), 15027. https://doi.org/10.1038/s41598-024-65691-0

[8] Cuomo, S., Di Cola, V.S., Giampaolo, F., Rozza, G., Raissi, M., Piccialli, F. Scientific machine learning through physics-informed neural networks: Where we are and what's next. Journal of Scientific Computing, 2022, 92(3), 88. https://doi.org/10.1007/s10915-022-01939-z

[9] Duan, S., Yu, S., Principe, J. Modularizing deep learning via pairwise learning with kernels. IEEE Transactions on Neural Networks and Learning Systems, ArXiv.. https://doi.org/10.48550/arXiv.2005.05541

[10] Raissi, M., Perdikaris, P., Karniadakis, G.E. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 2019, 378, 686-707. https://doi.org/10.1016/j.jcp.2018.10.045

[11] Beck, C., Weinan, E., Jentzen, A. Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations. Journal of Nonlinear Science, 2019, 29(4), 1563-1619. https://doi.org/10.1007/s00332-018-9525-3

[12] De Ryck, T., Mishra, S. Generic bounds on the approximation error for physics-informed (and) operator learning. Advances in Neural Information Processing Systems, 2022, 35, 10945-10958. https://doi.org/10.48550/arXiv.2205.11393

[13] Lu, L., Jin, P., Pang, G., Zhang, Z., Karniadakis, G.E. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 2021, 3(3), 218-229. https://doi.org/10.1038/s42256-021-00302-5

[14] Karniadakis, G.E., Kevrekidis, I.G., Lu, L., Perdikaris, P., Wang, S., Yang, L. Physics-informed machine learning. Nature Reviews Physics, 2021, 3(6), 422-440. https://doi.org/10.1038/s42254-021-00314-5

[15] Mishra, S., Molinaro, R. Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for PDEs. IMA Journal of Numerical Analysis, 2022, 42(2), 981-1022. https://doi.org/10.1093/imanum/drab032

[16] Berner, J., Grohs, P., Jentzen, A. Analysis of the generalization error: Empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black–Scholes partial differential equations. SIAM Journal on Mathematics of Data Science, 2020, 2(3), 631-657. https://doi.org/10.1137/19M125649X

[17] Haghighat, E., Raissi, M., Moure, A., Gomez, H., Juanes, R. A physics-informed deep learning framework for inversion and surrogate modeling in solid mechanics. Computer Methods in Applied Mechanics and Engineering, 2021, 379, 113741. https://doi.org/10.1016/j.cma.2021.113741

[18] Beck, C., Becker, S., Grohs, P., Jaafari, N., Jentzen, A. Solving the Kolmogorov PDE by means of deep learning. Journal of Scientific Computing, 2021, 88(3), 73. https://doi.org/10.1007/s10915-021-01590-0

[19] Sirignano, J., Spiliopoulos, K. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 2018, 375, 1339-1364. https://doi.org/10.1016/j.jcp.2018.08.029

[20] Bengio, Y., Lodi, A., Prouvost, A. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research, 2021, 290(2), 405-421. https://doi.org/10.1016/j.ejor.2020.07.063

[21] Rathi, N., Chakraborty, I., Kosta, A., Sengupta, A., Ankit, A., Panda, P., Roy, K. Exploring neuromorphic computing based on spiking neural networks: Algorithms to hardware. ACM Computing Surveys, 2023, 55(12), 1-49. https://doi.org/10.1145/3571155

[22] Hu, Q., Zhang, H., Gao, F., Xing, C., An, J. Analysis on the number of linear regions of piecewise linear neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(2), 644-653. https://doi.org/10.1109/TNNLS.2020.3028431




DOI: http://dx.doi.org/10.30970/eli.33.2

Refbacks

  • There are currently no refbacks.