This is a university project at TU Vienna to create a neural network hardware accelerator with an FPGA.
The network is designed and trained using Pytorch and Keras in Python.
Using Xilinx Vivado the Neural Network is implemented on Digilent Zedboard featuring a Zynq-7000 ARM/FPGA SoC.
The FPGA-Net can be controlled via a Webinterface based on Python Flask.
For more details see the Specification Document
Baischer Lukas, Leitner Anton, Kulnik Benjamin, Marschner Stefan, Cerv Miha
net: Here is everything stored that is relevant for training and designing the network. Pretrained netork parameters are available for torch, keras and in numpy format.
vivado:
This is the main project folder where the VHDL implementaiton of the neural network is stored. Additionally the linux driver software source files are stored in vivado/Software. For compiling and running the testbenches ghdl 0.3.7 and Vivado 2017.4 have been used. Additional python is needed to generate some of the source files or to run some
testbenches.
python:
This folder contains support software to verify the network. All basic network operations like 2d convolutions and poolingis reprogrammed. This is not necessary but helped to understand the internal functioning of the opaque interfaces of keras and pytorch.
webapp:
Here the Web app control software is stored. It is based on Flask and the LTEAdmin. Here images can be upladed
the netork evaluates it and sends back the result.
tex:
Contains the documentation of the project in Latex format. A precompiled PDF is available.
data:
Contains project data, e.g. the MNIST dataset.
+----------------------------+ +-----------------------------+
| CPU +-----------------+ | | |
| | | | | |
| | Flask | | | |
| | | | | |
| +------+-+--------+ | | |
| ^ | | | |
| | | | | FPGA |
| | v | | Eggnet Implementation |
| +------+-+--------+ | | |
| | +----->+ |
| | EggNet Driver | | | |
| | +<-----+ |
| +-----------------+ | | |
+----------------------------+ +-----------------------------+
- Train a Neural Network using Python
- Verfiy all the calculations
- Prepare Linux for the Zedboard
- Write the basic Neural Network Operations in VHDL
- Write a custom driver for the FPGA
- Setup FPGA to communicate with PC or use embedded Linux
- Verfiy VHDL implementation
- Optimize Network: Fixed Point
- Optimize Network: Parallelization and Pipelining
- Install driver, python and setup software on the zedboard
Other nice to have features, which will be tried to implement, if there is enough time.
- Webinterface for Easy Control
- Backpropagation on FPGA
- Custom Pytorch Extension to use the FPGA from within Torch
- CI Pipeline for VHDL/Vivado Build & Tests
For more details see the Specification Document
- Vivado 2017.4, for creating the FPGA implemenation
- Python, >3.6 + packages in
python/requirements.txt - C11 compiler for the Python Swig Extension
- GHDL: For VHDL testbench simulation checking
To accelerate certain neural network functions (conv2d, pool, etc.) those are reprogrammed in C and wrapped via SWIG.
See python/README.md for more details on this topic.
Open Vivado and select: Execute Tcl Script. Navigate to vivado/project. If problems occur use the cleanup script
in the folder (either cleanup.sh for Linux/Unix or cleanup.cmd for Windows). The bitstream can then be generated in
Vivado as usual.
First the python wrapper must be created using swig. Make sure SWIG >4.0 is installed and run the commands inside a
terminal:
cd projeckt/Software/eggnet
swig -python eggnet.iThis should create two files, eggnet_wrap.c and EggnetDriverCore.py. The first one must then be build togehter
with the rest of the project using the Vivado SDK or CMake.
Backpropagation in CNNs:
- https://jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/
- https://medium.com/the-bioinformatics-press/only-numpy-understanding-back-propagation-for-max-pooling-layer-in-multi-layer-cnn-with-example-f7be891ee4b4
- A guide to convolution arithmetic for deep learning: https://arxiv.org/pdf/1603.07285.pdf
Vanishing Gradients Problem:
Large-scale Learning with SVM and Convolutional Nets for Generic Object Categorization: