Series-parallel architecture for the FPGA implementation of neural networks trainable in real-time using the error backpropagation algorithm

To date, several Field-Programmed Gate Array (FPGA) implementable computational architectures have been proposed that can be used for neural network training in real-time by the backpropagation algorithm. However, they are intended for small neural networks or have a significant reduction in maximum clock frequency as network sizes increase. The novelty of this work lies in addressing the problems of ensuring a predictable maximum clock frequency and minimizing its degradation when scaling the computational architecture. The proposed architecture solves these problems at the level of computational organization. The architecture comprises an array of computational blocks which are based on FPGA digital signal processing blocks and perform most computations in parallel. The architecture also contains the shared block that sequentially processes the computation results received from the array blocks. The equations were derived showing that the latency of computations increases linearly with neural network sizes. After a computational block instance, the shared block and neural networks containing various numbers of computational blocks had been implemented on the FPGA, their timing characteristics were assessed. It has been determined that the data path delays of the buses connecting the shared block with the array blocks are the primary factors constraining the maximum clock frequencies of neural networks. When the number of the array blocks lies in the range 3–240, the maximum clock frequency is from 112 down to 77 MHz. Compared to the closest counterpart, the critical paths in the proposed architecture are shortened because some computations are transferred to the sequential mode; however, this transfer may increase the latency of calculating the local gradients of the hidden layers neurons. When the number of the array computational blocks grows from 3 to 128, the maximum clock frequency decreases by 27 % compared to 52 % for the closest counterpart. Growing the number of computational blocks in the proposed architecture from 128 to 240 reduces the maximum clock frequency by no more than 5 %. FPGA-based neural networks of the proposed architecture are suitable for object tracking and system identification, which are typical applications of neural networks trained in real-time mode.

1. Haykin S. Neural networks: A Comprehensive Foundation. Prentice Hall, 1998. 842 p.

2. Zhao G., Zhang P., Ma G., Xiao W. System identification of the nonlinear residual errors of an industrial robot using massive measurements. Robotics and Computer-Integrated Manufacturing, 2019, vol. 59, pp. 104–114. https://doi.org/10.1016/j.rcim.2019.03.007

3. Nelles O. Nonlinear System Identification: from Classical Approaches to Neural Networks, Fuzzy Models, and Gaussian Processes. Springer, 2020, 1253 p. https://doi.org/10.1007/978-3-030-47439-3

4. Han D., Yoo H.-J. On-Chip Training NPU-Algorithm, Architecture and SoC Design. Springer, 2023, 260 p. https://doi.org/10.1007/978- 3-031-34237-0

5. Han D., Lee J., Lee J., Choi S., Yoo H.-J. A 141.4 mW low-power online deep neural network training processor for real-time object tracking in mobile devices. Proc. of the IEEE International Symposium on Circuits and Systems (ISCAS), 2018, pp. 1–5. https://doi.org/10.1109/iscas.2018.8351398

6. Varagula J., Kulproma P., Itob T. Object detection method in traffic by on-board computer vision with time delay neural network. Procedia Computer Science, 2017, vol. 112, pp. 127–136. https://doi.org/10.1016/j.procs.2017.08.185

7. Heidaryan M., Karimi G. FPGA implementation of two multilayer perceptron neural network in cascade for efficient real time hand gestures tracking. Microprocessors and Microsystems, 2023, vol. 100, pp. 104849. https://doi.org/10.1016/j.micpro.2023.104849

8. Seng K.P., Ang L.M. Embedded intelligence: State-of-the-art and research challenges. IEEE Access, 2022, vol. 10, pp. 59236–59258. https://doi.org/10.1109/access.2022.3175574

9. Ortega-Zamorano F., Jerez J.M., Munoz D.U., Luque-Baena R.M., Franco L. Efficient implementation of the backpropagation algorithm in FPGAs and microcontrollers. IEEE Transactions on Neural Networks and Learning Systems, 2015, vol. 27, no. 9, pp. 1840–1850. https://doi.org/10.1109/tnnls.2015.2460991

10. Ushenina I.V., Danilov E.A. FPGA implementation of the artifical neuron module for series-parallel architectures of feedforward neural networks. Digital Signal Processing, 2025, no. 1, pp. 78–84. (in Russian)

11. Zhang M.J., Garcia S., Terre M. Real-time fast learning hardware implementation. International Journal for Simulation and Multidisciplinary Design Optimization, 2023, vol. 14, pp. 1. https://doi.org/10.1051/smdo/2023001

12. Tisan A., Chin J. An end-user platform for FPGA-based design and rapid prototyping of feedforward artificial neural networks with onchip backpropagation learning. IEEE Transactions on Industrial Informatics, 2016, vol. 12, no. 3, pp. 1124–1133. https://doi.org/10.1109/tii.2016.2555936

13. Zhang Z., Wang G., Wang K., Gan B., Chen G. Efficient on-chip learning of multi-layer perceptron based on neuron multiplexing method. Electronics, 2023, vol. 12, no. 17, pp. 3607. https://doi.org/10.3390/electronics12173607

14. Holt J.L., Hwang J.-N. Finite precision error analysis of neural network hardware implementations. IEEE Transactions on Computers, 1993, vol. 42, no. 3, pp. 281–290. https://doi.org/10.1109/12.210171

15. Ushenina I.V. Realization of the sigmoid activation function for neural networks on current FPGAs by the table-driven method. Tomsk State University Journal of Control and Computer Science, 2024, no. 69, pp. 124–133. (in Russian). https://doi.org/10.17223/19988605/69/13