48 amdgpu 21.50 plugin (#57)

* Added AMDGPU driver 21.50.2

* Added unified plugin creation  script for ROCr OpenCL and Legacy OpenCL

* Introduced directory for enabled plugins

* Added filter for pre-releases to Coinbootmaker when using default `latest` release

* Enabled optional usage of upstream AMD GPU firmware

* Excluded /boot directory from plugins

* Added well-known yet undocumented OpenCL variables

* Enabled setting the worker name via the environment

* Updated Team Red Miner to v0.9.4.2

* Reworked start of Team Red Miner in Systemd unit by calling exec

VERSIONING.md

@@ -0,0 +1,19 @@
+# Versioning of Coinboot
+
+## Scheme
+
+Versioning uses [CalVer](https://calver.org/) with the format: 
+
+**0Y.0M.0D-modifier**
+
+**0Y**        - Zero-padded year - 06, 16, 106  
+**0M**        - Zero-padded month - 01, 02 ... 11, 12  
+**0D**        - Zero-padded day - 01, 02 ... 30, 31  
+**-modifier**   - An optional text tag, such as "dev", "alpha", "beta", "rc1"  
+
+
+## Examples
+
+**22.05.01-rc0**  
+
+**22.01.30**

coinbootmaker/coinbootmaker

@@ -75,7 +75,7 @@ RELEASE=${RELEASE:-latest}
 if [ $RELEASE = latest ]; then
   RESPONSE=$($CURL --silent "https://api.github.com/repos/${GITHUB_REPO}/tags")
   sleep 5
-  while ! TAG=$(echo $RESPONSE | jq -r  '[ .[].name | select(test("^pre.*") | not) ] | sort | last'); do
+  while ! TAG=$(echo $RESPONSE | jq -r '[ .[].name | select(test("^pre.*") | not) ] | sort | last'  ); do
     echo "Calling the Github API has failed, repeat ..."
     RESPONSE=$($CURL --silent "https://api.github.com/repos/${GITHUB_REPO}/tags")
     sleep 5
@@ -169,6 +169,8 @@ sudo tee etc/hosts << EOF 1> /dev/null
 127.0.1.1 coinbootmaker
 EOF
 
+[ -d $BASEDIR/cache/apt ] || mkdir $BASEDIR/cache/apt
+
 cd $LOWER
 # Generate container spec file config.json
 # We have to manipulate the config.json created by 'runc spec'.
@@ -384,6 +386,15 @@ sudo tee ./config.json << EOF 1> /dev/null
                                 "bind",
                                 "ro"
                         ]
+                },
+                {
+                        "destination": "/var/cache/apt",
+                        "type": "none",
+                        "source": "$BASEDIR/cache/apt",
+                        "options": [
+                                "bind",
+                                "rw"
+                        ]
                 }
         ],
         "linux": {

debirf/scripts/create_plugin.py

@@ -41,21 +41,22 @@
 PLUGIN_DIR = "/mnt/plugin/rootfs"
 
 EXCLUDE = (
+    ".*__pycache__.*",
+    ".wget-hsts",
+    "/boot",
     "/dev/",
     "/proc/",
     "/run/",
     "/sys/",
     "/tmp/",
-    "/usr/src",
     "/usr/include",
     "/usr/share/dbus-1/system-services",
+    "/usr/src",
     "/vagrant",
     "/var/cache",
     "/var/lib/apt/lists",
     "/var/lib/dpkg/[^info]",
     "/var/log",
-    ".*__pycache__.*",
-    ".wget-hsts",
     r".*\.cache",
 )
 
@@ -135,8 +136,6 @@ def main(arguments):
         archive_name = arguments["<plugin_name>"] + ".tar.gz"
 
         create_tar_archive(archive_name, files_for_plugin_archive)
-
-        print("------------------------------------")
 
         print("------------------------------------")
 

plugins/README.md

@@ -20,7 +20,7 @@ For more information how to boot your machines with Coinboot visit: https://coin
 
 ## Requirements
 
-* Coinboot with kernel `5.4.0-58-generic`
+* Coinboot with kernel `5.11.0-46-generic`
 
 ### If you want to build plugins on your own: 
 
@@ -42,7 +42,7 @@ into the `plugins` directory of your Coinboot setup.
 To build Coinboot plugins on your own use `coinbootmaker`.
 
 ```
-Usage: coinbootmaker [-i] [-l] [-h] [-p <plugin name> <path to initramfs>]
+Usage: coinbootmaker [-i] [-l] [-h] [-p <plugin name>]
 
 -i              Interactive mode - opens a shell in the build environment
 -p <file name>  Plugin to build
@@ -53,20 +53,23 @@ Usage: coinbootmaker [-i] [-l] [-h] [-p <plugin name> <path to initramfs>]
 For example:
 
 ```
-$ ./coinbootmaker -p ethminer /tmp/coinboot-initramfs-4.15.0-43-generic  
+$ ./coinbootmaker -p ethminer
 ```
 
-`coinbootmaker` takes a path to a Coinboot-Initramfs to create an environment for building the plugins  
+`coinbootmaker` uses a Coinboot RootFS release to create an environment for building the plugins  
 by converting the given Initramfs into a Container image and run it.  
 The plugin creation script located at `src` is executed in that `coinbootmaker` container and the resulting  
-plugin archive is written to the `build` directory.
+plugin archive is written to the `build` directory. 
+#### Setting a Release
+You can specify the used Coinboot RootFS be setting the `RELEASE` enviroment variable.  
+The default value is `RELEASE=latetst` and will pulll the latest release of the Coinboot RootFS.  
 
 ### Run `coinbootmaker` interactivly (`-i`)
 
 For developing and debugging 
 
 ```
-$ ./coinbootmaker -i /tmp/coinboot-initramfs-4.15.0-43-generic 
+$ ./coinbootmaker -i
 ```
 
 * You are entering the build environment
@@ -77,7 +80,7 @@ $ ./coinbootmaker -i /tmp/coinboot-initramfs-4.15.0-43-generic
 
 * When your are done: Execute `$ create_plugin.py finish <name-of-your-plugin>`
 
-* Place the created plugin archive into `./plugins` on the host where you run the Coinboot Docker container
+* Place the created plugin archive into the directory `./plugins/enabled` on the host where you run the Coinboot Docker container
 
 Up on the next boot the changes your made in your plugin are ready to be used on your Coinboot machines!
 
@@ -98,5 +101,5 @@ [email protected]
 
 ## Contribution
 
-Fork this repo. 
+To contribute your own plugins fork this repo. 
 Make a pull request to this repo. 

plugins/README_template.md

@@ -12,7 +12,7 @@ For more information how to boot your machines with Coinboot visit: https://coin
 
 ## Requirements
 
-* Coinboot with kernel `5.4.0-58-generic`
+* Coinboot with kernel `5.11.0-46-generic`
 
 ### If you want to build plugins on your own: 
 
@@ -34,7 +34,7 @@ into the `plugins` directory of your Coinboot setup.
 To build Coinboot plugins on your own use `coinbootmaker`.
 
 ```
-Usage: coinbootmaker [-i] [-l] [-h] [-p <plugin name> <path to initramfs>]
+Usage: coinbootmaker [-i] [-l] [-h] [-p <plugin name>]
 
 -i              Interactive mode - opens a shell in the build environment
 -p <file name>  Plugin to build
@@ -45,20 +45,23 @@ Usage: coinbootmaker [-i] [-l] [-h] [-p <plugin name> <path to initramfs>]
 For example:
 
 ```
-$ ./coinbootmaker -p ethminer /tmp/coinboot-initramfs-4.15.0-43-generic  
+$ ./coinbootmaker -p ethminer
 ```
 
-`coinbootmaker` takes a path to a Coinboot-Initramfs to create an environment for building the plugins  
+`coinbootmaker` uses a Coinboot RootFS release to create an environment for building the plugins  
 by converting the given Initramfs into a Container image and run it.  
 The plugin creation script located at `src` is executed in that `coinbootmaker` container and the resulting  
-plugin archive is written to the `build` directory.
+plugin archive is written to the `build` directory. 
+#### Setting a Release
+You can specify the used Coinboot RootFS be setting the `RELEASE` enviroment variable.  
+The default value is `RELEASE=latetst` and will pulll the latest release of the Coinboot RootFS.  
 
 ### Run `coinbootmaker` interactivly (`-i`)
 
 For developing and debugging 
 
 ```
-$ ./coinbootmaker -i /tmp/coinboot-initramfs-4.15.0-43-generic 
+$ ./coinbootmaker -i
 ```
 
 * You are entering the build environment
@@ -69,7 +72,7 @@ $ ./coinbootmaker -i /tmp/coinboot-initramfs-4.15.0-43-generic
 
 * When your are done: Execute `$ create_plugin.py finish <name-of-your-plugin>`
 
-* Place the created plugin archive into `./plugins` on the host where you run the Coinboot Docker container
+* Place the created plugin archive into the directory `./plugins/enabled` on the host where you run the Coinboot Docker container
 
 Up on the next boot the changes your made in your plugin are ready to be used on your Coinboot machines!
 
@@ -90,5 +93,5 @@ [email protected]
 
 ## Contribution
 
-Fork this repo. 
+To contribute your own plugins fork this repo. 
 Make a pull request to this repo. 

plugins/src/amdgpu/21.40.2.40502/amdgpu_navi.yaml

@@ -0,0 +1,152 @@
+---
+# Copyright (C) 2018, 2020 - 2022 Gunter Miegel coinboot.io
+#
+# This file is part of Coinboot.
+# This software may be modified and distributed under the terms
+# of the MIT license.  See the LICENSE file for details.
+#
+# Please notice even while this script is licensed with the
+# MIT license the software packaged by this script may be licensed by
+# an other license with different terms and conditions.
+# You have to agree to the license of the packaged software to use it.
+
+plugin: AMDGPU Polaris
+archive_name: amdgpu_navi_rocr_opencl
+version: 21.40.2.40502
+description: AMD Navi GPU (RX 5000/RX 6000 family) firmware and driver with support for ROCr OpenCL 2.1
+maintainer: Gunter Miegel <[email protected]>
+source: https://www.amd.com/en/support/kb/release-notes/rn-amdgpu-unified-linux-21-40-2
+run: |
+  set -e
+
+  sudo apt update
+
+  # Specfiy AMD Radeon GPU series codename.
+  # RX 400/500 series: polaris
+  # RX Vega series: vega
+  # RX 5000/6000 series: navi
+  GPU_CODENAME=navi
+  AMDGPU_INSTALL_PACKAGE=amdgpu-install_21.40.2.40502-1_all.deb
+  if [ ! -z $DEBIRF_KERNEL ]; then
+    KERNEL_RELEASE=$DEBIRF_KERNEL
+  fi
+
+  PACKAGES="wget linux-headers-$KERNEL_RELEASE"
+  # We don't install any xserver stuff, similar to amdgpu-pro installer
+  # '--compute'
+  #AMDGPU_OPENCL_PACKAGES='clinfo-amdgpu-pro opencl-orca-amdgpu-pro-icd'rocm-opencl-runtime
+  AMDGPU_OPENCL_PACKAGES='rocm-opencl-runtime libnuma1'
+  # The DKMS package is installed explictily to purge it without wiping the dependcies.
+  AMDGPU_DKMS_PACKAGE='amdgpu-dkms'
+  AMDGPU_CACHE_PATH=/mnt/cache/amdgpu
+  EXTRA_MODULES_PACKAGE="linux-modules-extra-$KERNEL_RELEASE"
+  EXTRA_MODULES_PACKAGE_FILE=$(apt-cache show $EXTRA_MODULES_PACKAGE | grep -oP $EXTRA_MODULES_PACKAGE.*deb)
+  UPSTREAM_FIRMWARE_ARCHIVE=linux-firmware-20211216.tar.gz
+
+  sudo apt-get install $PACKAGES --yes
+
+  if [ -d $AMDGPU_CACHE_PATH ]; then
+    mkdir -p $AMDGPU_CACHE_PATH
+  fi
+
+  # Let's cache the downloaded archive and the decompressed
+  # archive on the build host - time saving once more.
+  # Download is only allowed with this referer!
+  if ! [ -f "$AMDGPU_CACHE_PATH"/"$AMDGPU_INSTALL_PACKAGE" ]; then
+    wget --progress=bar:force \
+       https://repo.radeon.com/amdgpu-install/21.40.2/ubuntu/focal/"$AMDGPU_INSTALL_PACKAGE" -P $AMDGPU_CACHE_PATH
+  else
+    echo "$AMDGPU_INSTALL_PACKAGE is already in build cache."
+  fi
+
+  if ! [ -f "$DEBIRF_ROOT/tmp/$AMDGPU_INSTALL_PACKAGE" ]; then
+   cp -r "$AMDGPU_CACHE_PATH/$AMDGPU_INSTALL_PACKAGE" /tmp/
+  else
+    echo "$AMDGPU_INSTALL_PACKAGE is already copied to the chroot."
+  fi
+
+  if ! [ -f "$AMDGPU_CACHE_PATH"/"$UPSTREAM_FIRMWARE_ARCHIVE" ]; then
+    wget --progress=bar:force \
+       https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/$UPSTREAM_FIRMWARE_ARCHIVE -P $AMDGPU_CACHE_PATH
+  else
+    echo "$UPSTREAM_FIRMWARE_ARCHIVE is already in build cache."
+  fi
+
+  if ! [ -f "$DEBIRF_ROOT/tmp/$UPSTREAM_FIRMWARE_ARCHIVE" ]; then
+   cp -r "$AMDGPU_CACHE_PATH/$UPSTREAM_FIRMWARE_ARCHIVE" /tmp/
+  else
+    echo "$UPSTREAM_FIRMWARE_ARCHIVE is already copied to the chroot."
+  fi
+
+  # The amdgpu-pro-install script is not running properly non-interactive
+  # it uses 'apt-get install' and asks for (Y/n).
+  # So we install the packages on our own.
+  # 1. Create the local deb repository
+  # 2. Update the package list
+  # 3. Install the packages
+
+  sudo apt-get install --yes /tmp/$AMDGPU_INSTALL_PACKAGE
+
+  sudo  sed -i 's/^#deb /deb /' /etc/apt/sources.list.d/amdgpu-proprietary.list
+
+  sudo apt-get update
+
+
+  # TODO mock update-initramfs called by /var/lib/dpkg/info/amdgpu-dkms.postinst
+
+  sudo tee /usr/bin/update-initramfs << EOF
+  #!/usr/bin/bash
+
+  true
+  EOF
+
+  sudo chmod 755 /usr/bin/update-initramfs
+
+  # FIXME: /opt/amdgpu-pro/lib/x86_64-linux-gnu/libOpenCL.so.1 missing when installing
+  # $AMDGPU_DKMS_PACKAGE and $AMDGPU_OPENCL_PACKAGES at once.
+  # sudo apt-get install --yes $AMDGPU_OPENCL_PACKAGES $AMDGPU_DKMS_PACKAGE
+  sudo apt-get install --yes $AMDGPU_OPENCL_PACKAGES
+
+  #sudo apt-mark hold $AMDGPU_OPENCL_PACKAGES
+
+  sudo apt-get install --yes $AMDGPU_DKMS_PACKAGE
+
+  # Purge installed packages of this modules to free up the space.
+  # But keep the OpenCL related packages.
+
+  # The module amdgpu.ko requires iommu_v2.ko which comes with the linux-modules-extra-
+  sudo apt-get download "$EXTRA_MODULES_PACKAGE"
+
+  # dpkg-deb -x needs a full path!
+  sudo dpkg -x $PWD/$EXTRA_MODULES_PACKAGE_FILE /tmp/
+  sudo rm -rf $PWD/$EXTRA_MODULES_PACKAGE_FILE
+
+  sudo find /tmp/lib -name 'iommu_v2.ko' -exec install -vD {} /lib/modules/"$KERNEL_RELEASE"/kernel/drivers/iommu/amd/iommu_v2.ko \;
+
+  sudo rm -rf /tmp/lib
+
+  sudo cp -v /lib/modules/"$KERNEL_RELEASE"/updates/dkms/amd* /tmp/
+
+  sudo cp -Rv /lib/firmware /tmp/
+
+  sudo apt-get purge $PACKAGES --yes
+
+  # Additional packages which can be removed to save space.
+  sudo apt-get purge --yes --allow-remove-essential ^gcc-9 grub-common dialog
+  sudo apt autoremove --yes
+
+  dpkg -l|grep opencl
+
+  sudo cp -v /tmp/amd*.ko /lib/modules/"$KERNEL_RELEASE"/updates/dkms/
+
+  sudo mkdir -p /lib/firmware/updates/amdgpu
+  sudo find /tmp/firmware -name "$GPU_CODENAME*" -exec cp -v {} /lib/firmware/updates/amdgpu/ \;
+
+  UPSTREAM_FIRMWARE_DIRECTORY=$(basename -s '.tar.gz' $UPSTREAM_FIRMWARE_ARCHIVE)/amdgpu
+  sudo tar -xzvf /tmp/$UPSTREAM_FIRMWARE_ARCHIVE -C /tmp -P $UPSTREAM_FIRMWARE_DIRECTORY
+  sudo cp -rv /tmp/$UPSTREAM_FIRMWARE_DIRECTORY /lib/firmware/updates/
+
+  sudo depmod $KERNEL_RELEASE
+
+  # Remove mocked update-initramfs
+  sudo rm -v /usr/bin/update-initramfs