Adding in RKE2 code, modifying from K3s solution as a base

Dockerfile

@@ -0,0 +1,40 @@
+FROM nvcr.io/nvidia/pytorch:23.10-py3
+
+ARG NCCL_TESTS_VERSION=master
+RUN apt-get update -y
+RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --allow-unauthenticated    \
+    wget \
+    curl \
+    git \
+    gcc \
+    vim \
+    kmod   \
+    openssh-client   \
+    openssh-server   \
+    build-essential  \
+    curl  \
+    autoconf \
+    libtool \
+    gdb \
+    automake \
+    python3-distutils  \
+    cmake  \
+    apt-utils \
+    devscripts \
+    debhelper  \
+    libsubunit-dev \
+    check \
+    pkg-config
+
+RUN mkdir -p /var/run/sshd
+RUN sed -i 's/[ #]\(.*StrictHostKeyChecking \).*/ \1no/g' /etc/ssh/ssh_config && \
+    echo "    UserKnownHostsFile /dev/null" >> /etc/ssh/ssh_config &&  \
+    sed -i 's/#\(StrictModes \).*/\1no/g' /etc/ssh/sshd_config
+
+RUN git clone https://github.com/NVIDIA/nccl-tests.git /opt/nccl-tests \
+    && cd /opt/nccl-tests \
+    && git checkout ${NCCL_TESTS_VERSION} \
+    && make MPI=1 MPI_HOME=/opt/hpcx/ompi CUDA_HOME=/usr/local/cuda
+
+RUN wget -O "/opt/h100-80gb-sxm-ib.xml" "https://docs.crusoecloud.com/assets/files/h100-80gb-sxm-ib-b914fb7ebbe8cbad591dd43abb36d687.xml"
+ENV NCCL_TOPO_FILE=/opt/h100-80gb-sxm-ib.xml

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Crusoe Cloud
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,2 +1,87 @@
-# crusoe-ml-rke2
-Deploying an RKE2 cluster on Crusoe Cloud with instructions for Nvidia GPU / network driver installation
+# Rancher K3s Deployment on Crusoe Cloud
+
+## Known Issues
+
+If the Terraform below fails contact [email protected] for help.
+
+## Requirements
+
+- [Terraform CLI](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
+- [Crusoe Cloud ClI](https://docs.crusoecloud.com/quickstart/installing-the-cli/index.html)
+- [Crusoe Cloud Credentials](https://docs.crusoecloud.com/account-management/managing-api-keys)
+  - Either stored in `~/.crusoe/credentials` or exported as environment variables `CRUSOE_x`
+- []
+
+## Deployment
+
+To use as a module fill in the `variables` in your own `main.tf` file
+
+```
+module "crusoe" {
+  source = "github.com/crusoecloud/crusoe-ml-k3s"
+  ssh_privkey_path="</path/to/priv.key"
+  ssh_pubkey="<pub_key>"
+  worker_instance_type = "h100-80gb-sxm-ib.8x"
+  worker_image = "ubuntu22.04-nvidia-sxm-docker:latest"
+  worker_count = 2
+  ib_partition_id = "6dcef748-dc30-49d8-9a0b-6ac87a27b4f8"
+  headnode_instance_type="c1a.8x"
+  deploy_location = "us-east1-a"
+  # extra variables here
+}
+```
+
+To use from this directory, fill in the `variables` in a `terraform.tfvars` file
+
+```
+ssh_privkey_path="</path/to/priv.key"
+ssh_pubkey="<pub_key>"
+worker_instance_type = "h100-80gb-sxm-ib.8x"
+worker_image = "ubuntu22.04-nvidia-sxm-docker:latest"
+worker_count = 2
+ib_partition_id = "6dcef748-dc30-49d8-9a0b-6ac87a27b4f8"
+headnode_instance_type="c1a.8x"
+deploy_location = "us-east1-a"
+# extra variables here
+```
+
+And then apply, to provision resources
+
+```
+terraform init
+terraform plan
+terraform apply
+```
+
+## Accessing the Cluster
+
+Once the deployment is complete, you can access the cluster by copying the `kubeconfig` file from the headnode. Replace the 'server' address in the Kubeconfig with that of your load balancer (or control plane node when deploying single control plane node configurations). 
+
+```bash
+k3s_endpoint=$(terraform output -raw k3-ingress-instance_public_ip)
+headnode_endpoint=$(terraform output -raw k3-headnode-instance_public_ip)
+scp -i $TF_VAR_ssh_privkey_path "root@${headnode_endpoint}:/etc/rancher/k3s/k3s.yaml" ./kubeconfig
+# change the endpoint to the kubectl endpoint
+sed -i '' "s/127.0.0.1/${k3s_endpoint}/g" ./kubeconfig
+# rename the context (optional)
+sed -i '' "s/default/crusoe/g" ./kubeconfig
+export KUBECONFIG="$(pwd)/kubeconfig"
+```
+
+## Nvidia GPU Support
+
+For nodes with Nvidia GPUs, you can run the following commands to install the Nvidia GPU operators (along with the Network operator when using IB-enabled nodes) to ensure they are available for use by pods provisioned in the cluster. 
+
+```bash
+helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
+helm repo update
+helm install --wait --generate-name -n gpu-operator --create-namespace nvidia/gpu-operator --set driver.rdma.enabled=true --set driver.rdma.useHostMofed=true
+helm install network-operator nvidia/network-operator -n nvidia-network-operator --create-namespace -f ./gpu-operator/values.yaml --wait
+```
+
+To test the GPU infiniband speeds, you can run the following commands.
+
+```bash
+kubectl apply -f https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.4.0/deploy/v2beta1/mpi-operator.yaml
+kubectl apply -f examples/nccl-test.yaml
+```