IMPLEMENTATION OF SIGMOID ACTIVATION FUNCTIONS ON FPGA FOR NEURAL NETWORKS

Abstract

The paper presents a literature review of the main methods of neural networks activation function approximation (sigmoidal function, hyperbolic tangent). The attention is focused on the accuracy and speed of the methods of approximation of the activation function and the advantages and disadvantages of these methods in their hardware implementation on the FPGA.
The mathematical descriptions of the sigmoidal function, the hyperbolic tangent and their derivatives are considered. The ratios for calculating sigmoidal functions with negative arguments are given. In order to evaluate the accuracy of the approximation of the activation functions and their derivatives, the mean absolute and mean maximum errors were used. The conversion of valid input data into an integer format is achieved by its multiplication by 210 and cutting off the fractional part. In the implementation, the following format of representation of real numbers is used: 16 bits, 1 sign bit and 15 bits for storing the resulting integer number. Negative numbers are presented in two's complement.
Three methods of approximation of sigmoidal function are considered. The first one is a piecewise linear approximation. A nonlinear function with positive arguments is replaced by four straight lines. The second method is the approximation by the second-order polynomial. A system of equations for calculating a, b and c coefficients for polynomials is obtained and values of these coefficients are calculated. The third method is the approximation by the second-order polynomial with a simplified expression. A comparison is made for the accuracy of approximations by three methods of sigmoidal function and its derivatives. It is noted that the errors of the approximation of the sigmoidal function are smaller than the errors of approximation of its derivatives. For the considered approximation methods, integer expressions were obtained, according to which the structural schemes of the implementation of the sigmoidal function on the FPGA were developed. The main elements of these schemes are registers, comparators, multipliers, adders, subtraction devices and three-state buffers.
Digital hardware implementation of sigmoid activation functions was performed in VHDL language in the Quartus II development environment using elements from standard library. Devices with the functiionality of sigmoidal function element are realized on FPGA EP3C16F484C6 of Cyclone III family. The simulation of the device calculating the sigmoidal function FA_Sigm_N3 in the time domain for positive and negative input data was carried out. The diagram of the device and its description and hardware resources necessary for the realization and calculation time are presented. The hardware realization of the sigmoid activation function on the DE0 stand was performed and the testing of its work was carried out.

Keywords: FPGA, VHDL, activation function, sigmoid function, piecewise linear approximation, mean absolute error, maximum absolute error, hardware resources FPGA hardware costs