build_dataflow_cfg hang during step_set_fifo_depths

[ATchelet]

Hi,

I'm trying to build my Brevitas trained Tiny YOLOv2 network in FINN.
I'm using DataflowBuildConfig() and build_dataflow_cfg() inside a Jupyter Notebook to build the network and the build is getting stuck on the step_set_fifo_depths.

I tried to see what the framework is currently doing and all I can see is that the last file modified was pyverilator_ipstitched/Vfinn_design_wrapper. I tried to see if there's any process related to it in htop, but I can't see any long running process except the notebook in the Docker container.

Anyone can help? I can provide further details if needed. I'm just not sure what would be useful.

Thanks in advance,
Alon

[mmrahorovic]

Hi Alon,

Great to hear you're using FINN for your project. I'll try my best to help you out with this.

The step_set_fifo_depths transform can sometimes take some time as it is running RTL simulation with pyverilator. Some questions that help are:

• Which branch of FINN are you using and which version of Vivado?
• How long has the step_set_fifo_depths been running?
• Did the IP-stitching phase complete (does the log file say it completed, which can be found under the main folder with the other build related files, specifically named something like vivado_stitch_proj_<random_string>).

[ATchelet]

Hi,

Thank you very much for the support!

• I was using the master branch (d1cc9cf) and Vivado and Vitis 2020.1
• It was running for 20h+ before I terminated it
• FINN continued to step_set_fifo_depths from it, and I no longer have the outputs as they were wiped (because they were stored in the /tmp/ folder). But I think the issue might lay there as after my problem I tried running the build with auto_fifo_step=False it failed due to a license issue. I am currently working on trying to figure it out as I was working on reporting an issue. That's where I currently am:
  • I originally, when I was still working on release 0.6 (c1e480d), added the license by adding DOCKER_EXEC+=" -e XILINXD_LICENSE_FILE=$XILINXD_LICENSE_FILE " to run-docker.sh and it worked.
  • But then when I moved to release 0.7 (d1cc9cf) it seems to have stopped working.
  • I saw on the FINN documentations that the license needs to be added to FINN_DOCKER_EXTRA instead so I moved it there but it still didn't work.
  • As part of opening the issue report, I checked out the latest development branch (5dd3f8c) and added my license variable again. It didn't finish building yet, but synthesised.

I will try again if the build passes and update here.

[mmrahorovic]

A couple of remarks:

• I assume your licensing issue is now resolved? Otherwise, I think you also need to mount the /path/to/licenses onto the Docker container. As an example, see: https://finn.readthedocs.io/en/latest/getting_started.html#vivado-vitis-license .
• Setting auto_fifo_step=False essentially skips the step_set_fifo_depths in the sense that it is assumed you are supplying the FIFO depths in the folding configuration file. If you did not supply the FIFO sizing there, your design will have no FIFOs and it will most likely impact the achievable throughput. So, in order to minimize the FIFO sizing and maximize the throughput, this particular step is required.
  Here is some more detail about the step I am referring to:

  finn/src/finn/transformation/fpgadataflow/set_fifo_depths.py

  Line 185 in d1cc9cf

  class InsertAndSetFIFODepths(Transformation):

• 20h+ sounds indeed like a lot, but depending on the size and folding of your network, it might require even more. You could verify whether the simulation hangs by adding a print statement in this while-loop (most likely the reason for a runtime of 20h+):

  finn/src/finn/transformation/fpgadataflow/set_fifo_depths.py

  Line 318 in d1cc9cf

  while ncycles > 0:

  For example, print ncycles every 100k cycles.
• Also, it would be helpful to estimate the cause of your problem if you could report the ncycles count. Essentially, given that model is your ONNX model before the step_set_fifo_depths, execute the following:

## Includes
from finn.transformation.fpgadataflow.annotate_cycles import AnnotateCycles
from finn.core.modelwrapper import ModelWrapper
from finn.analysis.fpgadataflow.dataflow_performance import dataflow_performance

model = model.transform(AnnotateCycles())
perf = model.analysis(dataflow_performance)
latency = perf["critical_path_cycles"]
max_cycles = perf["max_cycles"]
ncycles = int(latency+max_cycles)

What is the reported number of cycles?

Hope this helps you further! Please let me know if step_set_fifo_depths is working.

[ATchelet]

Hi,

Actually, the license worked and I got the output when step_set_fifo_depths=True.
I am now going to run the code with the additions you suggested.

The first suggestion where I print ncycles every 10K (I wanted a finer resolution) print first time at 61,200,000. I can also see from the elog that it takes the code about 4:10 minutes to run 10K cycles which would indeed mean this stage would take around 42.5 hours to complete.

Is there anything I can do to speed up the process?

Also, I am trying to compare the performance of the same network with different quantization levels (variation between W1A3 to W8A8). So the structure on the network is identical apart from the weights and activations bit width. Would it mean that I can use the same FIFO depths configuration for all of them or would I need to run the whole layers calculator for each one?

Thanks for all the support

[ATchelet]

Update after it finished:

So after 42 hours, step_set_fifo_depths finished.
But after it finished going through the ncycles it logged /workspace/finn/src/finn/transformation/fpgadataflow/set_fifo_depths.py:352: UserWarning: No output detected, calculated FIFO depths may not be correct.

By examining the generated final_hw_config.json, I can see that FIFO layers were added and their depth values are not all defaulted to 2 (as I think I've seen the default FIFO layer depths are).

Is there any demonstration showing the expected format to define FIFO layers through a json file so I could use these values in my variation of the same network with different quantization resolution?

Thanks again for all the support and thanks in advance for any future answers.

[mmrahorovic]

Hi @ATchelet,

No problem at all, I'm happy to help you out.

• Unfortunately, there is currently no way to ('quickly') speed up the process (step_set_fifo_depths=True).
• If the structure of the network remains the same (i.e. same convolution size, same number of channels, etc.) and if the folding factors remain the same, it sounds indeed reasonable to use the same FIFO depths configuration when the only thing you change is the quantization levels. However, I'm not 100% sure since I never tested this out. To be on the safe side, I would advise you to test it out. Try a different quantization level, run the step_set_fifo_depths=True, and see if the FIFOs are sized differently or exactly the same.
• The warning No output detected, calculated FIFO depths may not be correct comes from here. This is very odd and worth investigating further, perhaps some layer in your network stalls the process. I also cannot tell what is going wrong here. Could you make an issue out of this? Please also provide your initial ONNX file, all steps you run and other necessary files you think are needed for us to reproduce the problem.
• Could you share the final_hw_config.json? Are all FIFOs set to a very large number (2 ** 14?), or is the depth of only a few of them >2?
• You could use this as an example of how FIFO layers are defined in the folding_config.json file. Then, set auto_fifo_depths=False (like here and set folding_config_file to point to the folding_config.json file (like here).

Hope this helps you further!