Skip to content

Home

Jump to bottom
Greg Watson edited this page Nov 21, 2024 · 61 revisions

Welcome to the NetFPGA-PLUS-GW wiki!

This is a wiki for my work on NetFPGA-PLUS - see https://github.com/NetFPGA/NetFPGA-PLUS/wiki for the base repo.
This is just a list of various things I found useful when bringing up NetFPGA-PLUS.

Things I had to change (from 2021 version) to be able to run designs


/tools/settings - use new version of tools.
hw/lib/common/constraints/au250_general.xdc - Some clock paths had changed (Xilinx IP) and some clock definitions were missing.
hw/lib/common/hdl/top.v - use AU250 instead of AU280 (I have AU250)

Things I changed to make my life easier (optional)


hw/projects/reference_switch/hw/tcl/reference_switch.tcl - suppress subsequent warning messages after the first instance of the warning.

Errors encountered and possible fixes

  • CMAC licensing.
    You need a license for the 100G MAC. It is free but you must sign up and download it (You will need to sign in and complete a registration form for the license). Once you have the .lic file you must install it using vlm (information for this should be in the email accompanying the license).
    Make sure XILINXD_LICENSE_FILE variable points to the license file (.lic), or copy just the license part to the end of the current license file (sudo may be required)
    Run vlm and make sure that the path to the cmac_usplus license file is correct (it's in the XILINXD_LICENSE_FILE variable).
    Still have issues: look on the AMD Community Forum site under Installation And Licensing.
  • Error: application-specific initialization failed: couldn't load file "librdi_commontasks.so": libtinfo.so.5: cannot open shared object file: No such file or directory
    Fix: https://support.xilinx.com/s/article/76585?language=en_US
  • Error: Problem running nf_test.py: version issue between scapy library and the cryptography library:


    ERROR: Loading module scapy.layers.ipsec
    File "/usr/lib/python3/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 77, in _openssl_assert raise InternalError( cryptography.exceptions.InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ( [_OpenSSLErrorWithText(code=310378599, lib=37, reason=103, reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=310378599, lib=37, reas on=103, reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=126615813, lib=15, reason=786693, reason_text=b'error:078C0105:common libc


    Fix: use latest versions of the libraries. such as (Jan 2024) Python 3.10.12, pip 23.3.2, cryptograph 41.0.7

  • Error: I created a new library module in hw/lib/std but the simulation never located it.
    Fix: New modules in hw/lib need to be added to the top level Makefile in your $NFPLUS_FOLDER so that they are recompiled.
    Fix: Add the new module name to hw/projects/$NF_PROJECT_NAME/hw/tcl/$(NF_PROJECT_NAME)_sim.tcl
    Fix: Add the new module name to hw/projects/$NF_PROJECT_NAME/hw/tcl/export_registers.tcl (in DEF_LIST)
    Fix: Update the IP_ASSIGNMENT section in hw/projects/$NF_PROJECT_NAME/hw/tcl/$(NF_PROJECT_NAME)_defines.tcl

Useful Docs

Useful debug tools

  • pcimem (https://github.com/billfarrow/pcimem)
  • To verify that the Alveo board is registering with the PCIe bus enter the following Linux command in the terminal: 
    $ sudo lspci -vd 10ee:
  • Get status (and PCIe BDF ID) of the Alveo card (doesn't require netFPGA) 
    $ xbmgmt examine
  • To get the MAC addresses for the Alveo card: (replace 0000:01:00.0 with the PCIe BDF of your board) 
    sudo /opt/xilinx/xrt/bin/xbmgmt examine --report platform --device 0000:01:00.0
  • e.g. 
      Mac Address          : 00:0A:35:05:FE:80
                       : 00:0A:35:05:FE:81
  • Program the shell image - see https://docs.xilinx.com/r/en-US/ug1301-getting-started-guide-alveo-accelerator-cards/Programming-the-Shell-Partition-for-DFX-2RP-Platforms
  • Xilinx Alveo Debug Page: https://xilinx.github.io/Alveo-Cards/master/debugging/build/html/README.html

Programming the board

Per Xilinx KB article 71757 you may need to restore the board to its golden reference state under certain conditions:

  • Preparing to flash a different shell onto the card
  • Preparing to upgrade or change the version of XRT installed on the host
  • The card no longer appears on lspci after programming a custom image onto the card

Programming Flash via PCIE (not the USB cable)

  • Create .mcs file from .bit file. (Use tcl command from within Vivado)
  • Program board (command assume BDF is 01:00:0 and Au250 board bar offset is 0x40000. (NOTE: You may need to restore the golden image to the flash if the current flash image does not support direct PCIe writes to the flash.) 
     sudo /opt/xilinx/xrt/bin/xbflash.qspi --primary your_file.mcs --card 01:00.0 --bar 0x0 --bar-offset 0x40000

xbflash: Restoring Golden Image and then updating to current XRT

If you want to use xbflash to program the flash (not USB cable), but xbflash does not see the card then you may need to restore the original golden image and then bring the SC image back up date to match your installed SC version. This should only happen if the base image does not provide PCIe access to the flash memory.
Note: this is a two-step operation:

  • Restore the golden image to the base. This image is old and will not match current XRT software. See [KB article 71757](https://support.xilinx.com/s/article/71757?language=en_US) for info on how to do this. 
    e.g. $ sudo /opt/xilinx/xrt/bin/xbmgmt program --revert-to-golden --device 01:00.0
  • Update image to match Satellite Controller image (SC). See User Guide 1301 at https://docs.xilinx.com/r/en-US/ug1301-getting-started-guide-alveo-accelerator-cards/XRT-and-Deployment-Package-Installation-Procedures-on-Ubuntu 
    e.g. sudo /opt/xilinx/xrt/bin/xbmgmt program --base --device 01:00.0
    You can check the installed image version with 
     xbmgmt examine -r platform
After which you should see something like: 
$ sudo /opt/xilinx/xrt/bin/xbmgmt examine -d 01:00.0

[0000:01:00.0] : xilinx_u250_gen3x16_base_4


Flash properties
Type                 : spi
Serial Number        : 213204938004


Device properties
Type                 : u250
Name                 : AU250A64G
Config Mode          : 0x7
Max Power            : 225W


Flashable partitions running on FPGA
Platform             : xilinx_u250_gen3x16_base_4
SC Version           : 4.6.21
Platform UUID        : F8DAC62E-49D9-B0AA-E9FC-6F260D9D0DFB
Interface UUID       : 807A580E-5F50-7D48-484D-26C2217AA787


Flashable partitions installed in system
Platform             : xilinx_u250_gen3x16_base_4
SC Version           : 4.6.21
Platform UUID        : F8DAC62E-49D9-B0AA-E9FC-6F260D9D0DFB


Platform             : xilinx_u250_gen3x16_xdma_shell_4_1
Logic UUID           : 12C8FAFB-0632-499D-B1C0-C6676271B8A6
Interface UUID       : 807A580E-5F50-7D48-484D-26C2217AA787


Mac Address          : 00:0A:35:05:FE:80
: 00:0A:35:05:FE:81
Note that in the section "Flashable partitions installed in system" there is a base image (xilinx_u250_gen3x16_base_4) and a compatible shell image (xilinx_u250_gen3x16_xdma_shell_4_1). If the shell image is not compatible with the base then you will not see a shell image listed (see Xilinx User Guide UG1301).

Debugging ioctl issues

Useful post: https://stackoverflow.com/questions/31698765/which-driver-is-handling-my-ioctl

Clone this wiki locally