Driver blocked in wait mode on Alveo U250

[hleblevec]

Hi,
I'm trying to understand and learn how to use all the FINN flow on a custom network. I started by trying to implement the LowPrecisionLeNet that can be found on the Brevitas Github page as a toy network that would be fast to implement. I trained the network on CIFAR10. I've been able to generate the bitfile and pynq-based driver using FINN. However, I'm facing issues when trying to deploy the model and use the validate.py script. When executing the execute_on_buffer function, the program gets stuck indefinitely in the wait_until_finished call that's at the end of the function and I have to manually stop the program. In order to try to understand what happens, I trying deploying the tfc and cnv networks that can be found in the notebooks on the Alveo. In all cases, I used default parameters in FINN where I could, set the synthesis clock to 10 ns and did not define any target fps. I'm using the xilinx_u250_xdma_201830_2 shell version.
With tfc, I've been able to go through the whole test set without issues using a batch size of 100. With cnv, I can process a fraction of the dataset before it gets stuck in the wait statement the same way it does when using LeNet. The number of batches I can process depends on the batchsize: with a batchsize of 100 I can process 38 batches before it gets stuck, with a size of 10 it's 44, and with a size of 1 it's 98.

Have you ever experienced something like this before? Do you have any idea what could cause this?

[hleblevec]

Hi,

Let me update this and explain a little further.

I am trying to implement a ResNet-18 on Alveo U250 to later use it as the encoder of a U-Net.

In order to better understand FINN and Brevitas and how to use it I tried experimenting on light neural networks architectures that wouldn't take too long to process through the framework.
I started with a Low-Precision LeNet that I found here on the Brevitas Github repository.

To generate the hardware architecture I used the standard build dataflow of FINN and trained on MNIST.

I then tried to deploy it on the Alveo card using the generated PYNQ driver and the validate.py script.

However, the execution does not work as expected and the program gets stuck in a wait statement during the execution of the execute_on_buffers function (this line). It behaves as if the driver never received an output buffer from the DMA.

In order to try to understand and solve this issue I tried the following experiments:

• Processing and deploying the binary TFC architecture available here from the very beginning of the workflow (i.e. Brevitas training) to quickly verify if the problem was coming from the way I was building models. I successfully deployed the model and ran the validation script.

• I noticed that the LowPrecisionLeNet was using standard Pytorch MaxPool2d layer when there is a quantized equivalent in Brevitas. I changed that in the LeNet and trained again. I built the hardware and tried again to use the validation script. This time, I was able to process a couple of batches before getting stuck at the same level.

• I also noticed that LeNet uses biases in the convolution layers rather than Batchnorms when other Brevitas examples use the typical Conv-BN-ReLU blocks. Could this be the origin of the error? Does FINN support bias in convolutions?

• I assumed the problem lied in the architecture of the network or its description in Brevitas, so I started building a network layer by layer and building and deploying on each incrementation. So far, I have been able to validate two versions.
  The first one only has a convolution layer :

  class ToyNet_v1(Module):
      def __init__(self):
          super(ToyNet, self).__init__()
          self.quant_inp = qnn.QuantIdentity(
              bit_width=4, return_quant_tensor=True)
          self.conv1 = qnn.QuantConv2d(
              1, 3, 3, weight_bit_width=4, bias=False, return_quant_tensor=False, padding=1)
  

  In the second I added BatchNorm and ReLU layers:

  class ToyNet_v2(Module):
  def __init__(self):
      super(ToyNet, self).__init__()
      self.quant_inp = qnn.QuantIdentity(
          bit_width=4, return_quant_tensor=True)
      self.conv1 = qnn.QuantConv2d(
          1, 3, 3, weight_bit_width=4, bias=False, return_quant_tensor=False, padding=1)
      self.bn1= nn.BatchNorm2d(3)
      self.relu1 = qnn.QuantReLU(
          bit_width=4, return_quant_tensor=True)
  

  In the next version I tried to add a FC layer directly after, but came across an error during the building of the HLS layers: "ERROR: [XFORM 203-103] Array 'inputBuf.V' (/workspace/finn-hlslib/mvau.hpp:230): partitioned elements number (2352) has exeeded the threshold (1024), which may cause long run-time."
  With 2352 being exactly the number of input neurons of the FC layer (28 x 28 x 3), I assume this means that I am limited to a maximum of 1024 input neurons in FC layers. Is this some limitation of FINN or of Vivado HLS itself?

• In the meantime, I saw that there is a complete implementation of a ResNet-50 for Alveo U250 available here with the ONNX model and a custom dataflow. I successfully deployed the model on the board, and I am now wondering: would it be possible to have access to the Brevitas model? I haven't found it anywhere and I would like to use it as a reference to describe the ResNet-18.

Hope it clarifies the issue and questions, and I hope you can help.

Regards,

Hugo

[hleblevec]

Hi Felix,
Thank you for your reply.
I am investigating the points you mentioned. I can already git an answer to some of them.

• I am indeed using the last dev branch.

• I don't really understand what you mean by "setting the io_shape_dict since the driver code is generated automatically by FINN and so is this dictionary. I have, however, noted something weird in the values that are automatically set up. Let me explain: I continued my experimentation with building a DNN layer by layer. The next version was looking like this:

  class ToyNet_v3(Module):
      def __init__(self):
          super(ToyNet, self).__init__()
          self.quant_inp = qnn.QuantIdentity(
              bit_width=4, return_quant_tensor=True)
          self.conv1 = qnn.QuantConv2d(
              1, 3, 3, weight_bit_width=4, bias=False, return_quant_tensor=False, padding=1)
          self.bn1= nn.BatchNorm2d(3)
          self.relu1 = qnn.QuantReLU(
              bit_width=4, return_quant_tensor=True)
          self.pool = qnn.QuantMaxPool2d(kernel_size=2)
          self.fc1   = qnn.QuantLinear(
          14*14*3, 10, bias=True, bias_quant=BiasQuant, weight_bit_width=4, return_quant_tensor=True)
  

I've been able to process the network through FINN, but I am experiencing the same issue during the deployment and the program gets stuck in the execute_on_buffers functions. I then got the idea to reduce the input size by 2 on each dimension (so going from (1,28,28) to (1,14,14)) and removing the MaxPool2d layer. This time, I did not get the blocking issue and have been able to process 10000 resized images of MNIST trough the network on the Alveo without issue. It seems that removing the MaxPool layer solved the problem. Why do you think could be the reason behind this? Here the ONNX models of both the ToyNet_v3 and the version without max pooling (name ToyNet_v3_half): models.zip.

Regarding the io_shape_dict, the one for the ToyNet_v3 is:

io_shape_dict = {
    # FINN DataType for input and output tensors
    "idt" : [DataType['INT4']],
    "odt" : [DataType['INT24']],
    # shapes for input and output tensors (NHWC layout)
    "ishape_normal" : [(1, 28, 28, 1)],
    "oshape_normal" : [(1, 10)],
    # folded / packed shapes below depend on idt/odt and input/output
    # PE/SIMD parallelization settings -- these are calculated by the
    # FINN compiler.
    "ishape_folded" : [(1, 28, 28, 1, 1)],
    "oshape_folded" : [(1, 10, 1)],
    "ishape_packed" : [(1, 28, 28, 1, 1)],
    "oshape_packed" : [(1, 10, 3)],
    "input_dma_name" : ['idma0'],
    "output_dma_name" : ['odma0'],
    "number_of_external_weights": 0,
    "num_inputs" : 1,
    "num_outputs" : 1,
}

The one for the ToyNet_v3_half is the same except for the ishape_normal, ishape_folded and ishape_packed which have (14,14) instead of (28,28). This is making me wonder about several points:

• What is the difference between the "packed" and "folded" shapes?
• Why is oshape_packed (1, 10, 3) instead of (1, 10, 1)?
• Why is the output in INT24?
• How do I know these values match the accelerator and its folding configuration?

In addition, I am seeing a similar behaviour with the LowPrecisionLeNet:

io_shape_dict = {
    # FINN DataType for input and output tensors
    "idt" : [DataType['INT4']],
    "odt" : [DataType['INT16']],
    # shapes for input and output tensors (NHWC layout)
    "ishape_normal" : [(1, 32, 32, 3)],
    "oshape_normal" : [(1, 10)],
    # folded / packed shapes below depend on idt/odt and input/output
    # PE/SIMD parallelization settings -- these are calculated by the
    # FINN compiler.
    "ishape_folded" : [(1, 32, 32, 1, 3)],
    "oshape_folded" : [(1, 10, 1)],
    "ishape_packed" : [(1, 32, 32, 1, 3)],
    "oshape_packed" : [(1, 10, 2)],
    "input_dma_name" : ['idma0'],
    "output_dma_name" : ['odma0'],
    "number_of_external_weights": 0,
    "num_inputs" : 1,
    "num_outputs" : 1,

I am going to try the verification steps as you mentioned and I will give updates once I have compiled some information.

Thanks for helping me.

[hleblevec]

Hi,

I have conducted some new experiments to try to solve this case.

As you suggested, I tried going through the various verification steps included in the FINN flow. All of them have passed for the LeNet.
I also took a look at the waveforms generated by PyVerilator. I did not notice any strange behavior except for the very long delay before the first valid output from the design (several hundreds of clock cycles). Is this a normal behavior?
I leave the waveforms here in case you want to have a look: waveforms.zip

I also checked the timing reports in Vivado as you suggested, and there was no timing issues.

What do you think? Should I try to setup an ILA core and look at the signals directly during the run?

Hope you can help.

Regards,

Hugo

[seniorgoogles]

Hi,

I am also interested in solving the problem. At the point like Hugo I was so far also. Are there any approaches to solve the problem and follow up?

Kind regards,
Michael

[seniorgoogles]

Hi,

I am also interested in solving the problem. At the point like Hugo I was so far also. Are there any approaches to solve the problem and follow up?

Kind regards, Michael

Ok, got it. The runtime weights were not loaded, which caused the wait state. The runtime_weight_dir needs to be set correctly. If it is not set, no exception will be thrown.