Generated IP Block Behaves Differently in Vivado From Pytorch Model

[MichalVarsanyi]

Project Introduction

Hello everyone 👋,

I’m currently developing a demo that implements basic numeral recognition on an FPGA. My setup uses the Cmod A7 15T FPGA development board, which houses the AMD Artix-7 xc7a15tcpg236-1.

The interface consists of a 5x5 grid of buttons where a user "draws" a numeral. This grid serves as the input for a simple Neural Network (NN) that infers which digit was drawn. Below is an image of the board illustrating the intended use case:

I have created a GitHub repository with the source code here: MichalVarsanyi/FPGA-NN-Demo

---

Workflow

As a Windows user, I encountered limitations with certain hls4ml commands; specifically, .compile() and .build() do not function natively in this environment. To work around this, I used hls4ml to .write() the model and then manually executed the build process:

1. RTL Synthesis: Called vitis_hls -f build_prj.tcl to generate the RTL model.
2. IP Export: Created a custom export.tcl script to wrap the generated HDL files into a Vivado IP block.

open_project myproject_prj
open_solution solution1
export_design -format ip_catalog -description "My FPGA NN" -vendor "hls4ml"
exit

---

The Issue

The core issue is that the generated IP core produces seemingly random or incorrect inference results. Below is the pipeline visualization generated by the HLS model:

The following image shows the IP core instantiated in Vivado. The input dimensions match the model pipeline (16 bits * 25 inputs = 400 bits total), and the output dimensions are also consistent with the design.

During Vivado behavioral simulation, I send 400 bits to the input and monitor the output. I configured the output display settings to: Radix > Real Settings > Fixed Point > Signed with the binary point set after 20 bits (37-17=20).

The simulation results show significant inaccuracies. I looked at the largest logit at the output of the NN to make the inference prediction.

• Input 2 is inferred as 3
• Input 3 is inferred as 3 (Correct)
• Input 4 is inferred as 4 (Correct)
• Input 7 is inferred as 3
• Input 0 is inferred as 3

You can find the testbench used to generate this waveform here: FPGA_NN_TB.txt

---

Additional Information

To verify my toolchain, I compiled a simplified "Pass-Through" NN where a single 16-bit input is passed directly to a 16-bit output. This test worked perfectly, with the input successfully appearing at the output one clock cycle later.

Because the simple model works, I believe the vitis_hls conversion process itself is functional, but there may be an issue with how the multi-input data is being handled or interpreted by the full model.

I welcome any suggestions on why I may be seeing this behaviour :))
If you need any further files or logs, please let me know.
Thank you for the help!

[thesps]

This looks like a nice project!

So you see unexpected predictions after Vivado simulation. Is the Vivado simulation result consistent with the HLS C Simulation result? And is the HLS C Simulation result consistent with the trained NN? If you haven't gone through these steps yet, you may want to make sure that you get agreement at both of those stages before looking at the Vivado behavioural simulation in more detail. These will help to isolate whether the issue is related to the integration of the HLS IP with the interfaces, or the configuration of the hls4ml HLS NN. For instance, errors due to quantization of model parameters may cause disagreement already at the HLS C Simulation comparison with the trained NN.

[MichalVarsanyi]

Thank you for the reply! I’m still unable to get the simulation running (likely I’ll need to spend a bit more time looking at the C++ file), but in the meantime, I isolated the issue by building a small chain of neural nets for debugging. Sharing in case it helps others:

• First, I trained a simple NN where if the input is greater than 1, the output will be 1.5. This helped me confirm I am displaying the fixed-point numbers correctly.
• Second, I trained a 5x5 NN to simulate the grid of buttons I am working with. When I write 1 to the first row, first column, I get 1 in the output. If I write 1 in the first row, third column, I get 3 as an output. This type of indexing NN helped me verify that I am parsing the input correctly.

The root cause was that the model precision was too low, and I was indexing the inputs in a different order than expected. After increasing precision and fixing the input ordering, the model classified correctly in Vivado simulation.

Furthermore, @bo3z thank you very much for your reply as well! Unfortunately, I could not make two comments the "answer", but your input was equally helpful!

[bo3z]

Adding to @thesps comment - there's generally two sources of "wrong" outputs in Vivado simulation.

• All operations in hls4ml are done in fixed-point arithmetic. If your input model is floating point, the post-training quantisation by hls4ml may lead to differences in the output.
• The Vivado testbench itself may be incorrectly set-up. We found that in some versions of Vivado it's necessary to reverse the bits for each input variable.

To isolate your issue, I recommend:

• Running model "emulation" (i.e. just prediction using a normal C++ compiler), with model.predict(input_data) and comparing the output to your high-level NN (from PyTorch / Keras). If there's significant differences, consider using quantization-aware training (with for e.g., QKeras or HGQ) or increasing default_precision
• Do HLS simulation and co-simulation with some pre-defined input. During hls4ml model conversion you can specifiy an input and output NumPy arrays which will be used for running the testbench. The actual CSim / CoSim can be set in the build(...) function. If there are no differences to the above step, it confirms the HLS model is correctly compiled and the issue is likely in the data delivery to the DUT.