Add extractor deployment in jupyter notebook

jupyter/experiment_notebook.ipynb

@@ -353,4 +353,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 1
-}
+}

jupyter/terraform_notebook.ipynb

@@ -479,12 +479,17 @@
    },
    "outputs": [],
    "source": [
+    "###\n",
+    "### ! CHANGE ME\n",
+    "###\n",
+    "MAX_NUM_NODES=2\n",
+    "\n",
     "gcloud container clusters update $CLUSTER_NAME --node-pool $DEFAULT_POOL \\\n",
     "    --no-enable-autoscaling --region $REGION --quiet\n",
     "    \n",
     "# The high performance node will scale up automatically whenever the workloads are deployed\n",
     "gcloud container clusters update $CLUSTER_NAME --node-pool $EXPERIMENT_POOL \\\n",
-    "    --enable-autoscaling --min-nodes=0 --max-nodes=10 --region $REGION --quiet\n",
+    "    --enable-autoscaling --min-nodes=0 --max-nodes=$MAX_NUM_NODES --region $REGION --quiet\n",
     "\n",
     "gcloud container clusters resize $CLUSTER_NAME --node-pool $DEFAULT_POOL \\\n",
     "    --num-nodes 1 --region $REGION --quiet\n"
@@ -594,6 +599,88 @@
     "terraform -chdir=$TERRAFORM_DEPENDENCIES_DIR apply -auto-approve -var project_id=$PROJECT_ID"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "## Deploying extractor\n",
+    "\n",
+    "Lastly, we deploy the extractor pod, which also provides PVCs which can be used for artifact retrieval.\n",
+    "\n",
+    "Retrieval can be done by running\n",
+    "\n",
+    "```bash\n",
+    "EXTRACTOR_POD_NAME=$(kubectl get pods -n test -l \"app.kubernetes.io/name=fltk.extractor\" -o jsonpath=\"{.items[0].metadata.name}\")\n",
+    "kubectl cp -n test $EXTRACTOR_POD_NAME:/opt/federation-lab/logging ./logging\n",
+    "```\n",
+    "\n",
+    "For copying from the extractor path `/opt/federation-lab/logging` to a directory locally named `logging`.\n",
+    "\n",
+    "First build the docker container, following the instructions of the [readme](https://github.com/JMGaljaard/fltk-testbed#creating-and-uploading-docker-container).\n",
+    "\n",
+    "\n",
+    "N.B. Make sure to have setup a working authentication provider for docker, such that you can push to your repository."
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   },
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "Run this in a terminal in the content-root directory (so `fltk-testbed` if the project name was not altered).\n",
+    "```bash\n",
+    "python3 -m venv venv\n",
+    "source venv\n",
+    "pip3 install -r requirements-cpu.txt\n",
+    "python3 -m fltk extractor configs/example_cloud_experiment.json\n",
+    "```"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%% md\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "# Build the docker container with buildkit. Make sure you have Docker Desktop running on Windows/MacOS\n",
+    "DOCKER_BUILDKIT=1 docker build --platform linux/amd64 ../ --tag gcr.io/$PROJECT_ID/fltk\n",
+    "docker push gcr.io/<project-id>/fltk"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "# Install the extractor, and set the projectName to $PROJECT_ID.\n",
+    "# In case you get a warning regarding the namespace test, this means that the dependencies have not been properly installed.\n",
+    "# Make sure to check whether you have enough resources available, and re-run the installation of dependencies. (see above).\n",
+    "\n",
+    "# Deploy extractor, in test namespace with updated image reference (--set overwrites values from `fltk-values.yaml`).\n",
+    "helm install extractor ../charts/extractor -f ../charts/fltk-values.yaml --namespace test --set provider.projectName=$PROJECT_ID"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
   {
    "cell_type": "markdown",
    "metadata": {
@@ -728,4 +815,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 1
-}
+}