Skip to content

Commit

Permalink
Update sgutil to hdev
Browse files Browse the repository at this point in the history
  • Loading branch information
bo3z committed Dec 13, 2024
1 parent 6e3d30a commit 19cd54a
Show file tree
Hide file tree
Showing 5 changed files with 16 additions and 17 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ in the original build-directory as described before.
## Deploying on the ETHZ HACC-cluster
The ETHZ HACC is a premiere cluster for research in systems, architecture, and applications (https://github.com/fpgasystems/hacc/tree/main). Its hardware equipment provides the ideal environment to run Coyote-based experiments, since users can book up to 10 servers with U55C-accelerator cards connected via a fully switched 100G-network. User accounts for this platform can be obtained following the explanation on the previously cited homepage.

The interaction with the HACC-cluster can be simplified by using the sgutil-run time commands. They also allow to easily program the accelerator with a Coyote-bitstreamd and insert the driver. For this purpose, the scripts `util/program_hacc_local.sh` and `util/program_hacc_remote.sh` have been created. Under the assumption that the hardware-project has been created in `examples_hw/build` and the driver is already compiled in `driver`, the workflow should look like this:
The interaction with the HACC-cluster can be simplified by using the hdev-run time commands. They also allow to easily program the accelerator with a Coyote-bitstreamd and insert the driver. For this purpose, the scripts `util/program_hacc_local.sh` and `util/program_hacc_remote.sh` have been created. Under the assumption that the hardware-project has been created in `examples_hw/build` and the driver is already compiled in `driver`, the workflow should look like this:

~~~
$ bash util/program_hacc_local.sh examples_hw/build/bitstreams/cyt_top.bit driver/coyote_drv.ko
Expand Down
14 changes: 7 additions & 7 deletions docs/source/hacc/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,28 +17,28 @@ In terms of networking communication, each Alveo cards has two 100 Gbps interfac

For a more detailed look at how ETHZ-HACC is organized you can check out the following link: `ETHZ-HACC <https://github.com/fpgasystems/hacc/blob/main/docs/infrastructure.md#infrastructure>`_.

SGRT - Systems Group RunTime
hdev - HACC Development
====================================

Systems Group RunTime (SGRT) is a versatile RunTime software ready to be used on any AMD-compatible heterogeneous cluster.
HACC Development (hdev) is a versatile RunTime software ready to be used on any AMD-compatible heterogeneous cluster.

SGRT includes a command-line interpreter (CLI) and an API, both utilizing an intuitive device index to improve user workflow.
hdev includes a command-line interpreter (CLI) and an API, both utilizing an intuitive device index to improve user workflow.
The CLI simplifies infrastructure setup, validation, and device configuration, while the API streamlines accelerated application development, allowing users to focus on their primary objectives.

Using Coyote with SGRT
Using Coyote with hdev
------------------------

The SGRT provides a range of functions which can be used to quickly deploy Coyote on the HACC cluster.
The hdev provides a range of functions which can be used to quickly deploy Coyote on the HACC cluster.

To load the initial static Coyote image the following command can be used:

.. code-block::
sgutil program vivado -d <device_id>
hdev program vivado -d <device_id>
This will load the default bitstream obtained with the `static` example present in ``examples_hw``.
This static layer will thus be the same no matter where the bitstream is built and the shell layers can be swapped on the fly.

The above command will also handle the hot-plug protocol which rescannes the interconnect, thus no additional warm reboots are necessary,

The SGRT provides additional helper features that you can use, like validation and building scripts.
hdev provides additional helper features that you can use, like validation and building scripts.
6 changes: 3 additions & 3 deletions util/insmod_local.sh
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Script to set up IP and MAC address environment variables and insert the driver because environment is not initialized when executing through ssh from a remote server (environment is only initialized when MOTD is shown).

CLI_PATH=/opt/sgrt/cli
CLI_PATH=/opt/hdev/cli

IP_address=$($CLI_PATH/sgutil get network -d 1 | awk '$1 == "1:" {print $2}')
MAC_address=$($CLI_PATH/sgutil get network -d 1 | awk '$1 == "1:" {print $3}' | tr -d '()')
IP_address=$($CLI_PATH/hdev get network -d 1 | awk '$1 == "1:" {print $2}')
MAC_address=$($CLI_PATH/hdev get network -d 1 | awk '$1 == "1:" {print $3}' | tr -d '()')
qsfp_ip=$($CLI_PATH/common/address_to_hex IP $IP_address)
qsfp_mac=$($CLI_PATH/common/address_to_hex MAC $MAC_address)
echo "** IP_ADDRESS: $qsfp_ip"
Expand Down
8 changes: 4 additions & 4 deletions util/program_hacc_local.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,14 @@ fi


##
## Program the FPGA via the sgutil call - only work locally for the server that you are currently logged in to
## Program the FPGA via the hdev call - only work locally for the server that you are currently logged in to
##

if [ $PROGRAM_FPGA -eq 1 ]; then
echo "***"
echo "** Programming the FPGA with $BIT_PATH"
echo "***"
sgutil program vivado -b $BIT_PATH
hdev program vivado -b $BIT_PATH
echo "***"
echo "** FPGA programmed"
echo "***"
Expand All @@ -56,8 +56,8 @@ if [ $DRV_INSERT -eq 1 ]; then
echo "***"
echo "** IP_ADDRESS: $DEVICE_1_IP_ADDRESS_HEX_0"
echo "** MAC_ADDRESS: $DEVICE_1_MAC_ADDRESS_0"
sgutil program driver -i $DRV_PATH -p ip_addr=$DEVICE_1_IP_ADDRESS_HEX_0,mac_addr=$DEVICE_1_MAC_ADDRESS_0
# sgutil program driver -m $DRV_PATH
hdev program driver -i $DRV_PATH -p ip_addr=$DEVICE_1_IP_ADDRESS_HEX_0,mac_addr=$DEVICE_1_MAC_ADDRESS_0
# hdev program driver -m $DRV_PATH
echo "***"
echo "** Driver loaded "
echo "***"
Expand Down
3 changes: 1 addition & 2 deletions util/program_hacc_remote.sh
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,7 @@ if [ $DRV_INSERT -eq 1 ]; then

echo "*** Rescan PCIe ..."
echo " ** "
#parallel-ssh -H "$hostlist" -x '-tt' 'sudo /opt/sgrt/cli/program/pci_hot_plug "$(hostname -s)"'
parallel-ssh -H "$hostlist" -x '-tt' 'upstream_port=$(/opt/sgrt/cli/get/get_fpga_device_param 1 upstream_port) && root_port=$(/opt/sgrt/cli/get/get_fpga_device_param 1 root_port) && LinkCtl=$(/opt/sgrt/cli/get/get_fpga_device_param 1 LinkCtl) && sudo /opt/sgrt/cli/program/pci_hot_plug 1 $upstream_port $root_port $LinkCtl'
parallel-ssh -H "$hostlist" -x '-tt' 'upstream_port=$(/opt/hdev/cli/get/get_fpga_device_param 1 upstream_port) && root_port=$(/opt/hdev/cli/get/get_fpga_device_param 1 root_port) && LinkCtl=$(/opt/hdev/cli/get/get_fpga_device_param 1 LinkCtl) && sudo /opt/hdev/cli/program/pci_hot_plug 1 $upstream_port $root_port $LinkCtl'
# read -p "Hot-reset done. Press enter to load the driver or Ctrl-C to exit."

echo "*** Compiling the driver ..."
Expand Down

0 comments on commit 19cd54a

Please sign in to comment.