diff --git a/jupyter/experiment_notebook.ipynb b/jupyter/experiment_notebook.ipynb index 121d124d..55672c84 100644 --- a/jupyter/experiment_notebook.ipynb +++ b/jupyter/experiment_notebook.ipynb @@ -353,4 +353,4 @@ }, "nbformat": 4, "nbformat_minor": 1 -} +} \ No newline at end of file diff --git a/jupyter/terraform_notebook.ipynb b/jupyter/terraform_notebook.ipynb index 2d9f10b1..12c5134c 100644 --- a/jupyter/terraform_notebook.ipynb +++ b/jupyter/terraform_notebook.ipynb @@ -479,12 +479,17 @@ }, "outputs": [], "source": [ + "###\n", + "### ! CHANGE ME\n", + "###\n", + "MAX_NUM_NODES=2\n", + "\n", "gcloud container clusters update $CLUSTER_NAME --node-pool $DEFAULT_POOL \\\n", " --no-enable-autoscaling --region $REGION --quiet\n", " \n", "# The high performance node will scale up automatically whenever the workloads are deployed\n", "gcloud container clusters update $CLUSTER_NAME --node-pool $EXPERIMENT_POOL \\\n", - " --enable-autoscaling --min-nodes=0 --max-nodes=10 --region $REGION --quiet\n", + " --enable-autoscaling --min-nodes=0 --max-nodes=$MAX_NUM_NODES --region $REGION --quiet\n", "\n", "gcloud container clusters resize $CLUSTER_NAME --node-pool $DEFAULT_POOL \\\n", " --num-nodes 1 --region $REGION --quiet\n" @@ -594,6 +599,88 @@ "terraform -chdir=$TERRAFORM_DEPENDENCIES_DIR apply -auto-approve -var project_id=$PROJECT_ID" ] }, + { + "cell_type": "markdown", + "source": [ + "## Deploying extractor\n", + "\n", + "Lastly, we deploy the extractor pod, which also provides PVCs which can be used for artifact retrieval.\n", + "\n", + "Retrieval can be done by running\n", + "\n", + "```bash\n", + "EXTRACTOR_POD_NAME=$(kubectl get pods -n test -l \"app.kubernetes.io/name=fltk.extractor\" -o jsonpath=\"{.items[0].metadata.name}\")\n", + "kubectl cp -n test $EXTRACTOR_POD_NAME:/opt/federation-lab/logging ./logging\n", + "```\n", + "\n", + "For copying from the extractor path `/opt/federation-lab/logging` to a directory locally named `logging`.\n", + "\n", + "First build the docker container, following the instructions of the [readme](https://github.com/JMGaljaard/fltk-testbed#creating-and-uploading-docker-container).\n", + "\n", + "\n", + "N.B. Make sure to have setup a working authentication provider for docker, such that you can push to your repository." + ], + "metadata": { + "collapsed": false, + "pycharm": { + "name": "#%% md\n" + } + }, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "Run this in a terminal in the content-root directory (so `fltk-testbed` if the project name was not altered).\n", + "```bash\n", + "python3 -m venv venv\n", + "source venv\n", + "pip3 install -r requirements-cpu.txt\n", + "python3 -m fltk extractor configs/example_cloud_experiment.json\n", + "```" + ], + "metadata": { + "collapsed": false, + "pycharm": { + "name": "#%% md\n" + } + } + }, + { + "cell_type": "code", + "execution_count": null, + "outputs": [], + "source": [ + "# Build the docker container with buildkit. Make sure you have Docker Desktop running on Windows/MacOS\n", + "DOCKER_BUILDKIT=1 docker build --platform linux/amd64 ../ --tag gcr.io/$PROJECT_ID/fltk\n", + "docker push gcr.io//fltk" + ], + "metadata": { + "collapsed": false, + "pycharm": { + "name": "#%%\n" + } + } + }, + { + "cell_type": "code", + "execution_count": null, + "outputs": [], + "source": [ + "# Install the extractor, and set the projectName to $PROJECT_ID.\n", + "# In case you get a warning regarding the namespace test, this means that the dependencies have not been properly installed.\n", + "# Make sure to check whether you have enough resources available, and re-run the installation of dependencies. (see above).\n", + "\n", + "# Deploy extractor, in test namespace with updated image reference (--set overwrites values from `fltk-values.yaml`).\n", + "helm install extractor ../charts/extractor -f ../charts/fltk-values.yaml --namespace test --set provider.projectName=$PROJECT_ID" + ], + "metadata": { + "collapsed": false, + "pycharm": { + "name": "#%%\n" + } + } + }, { "cell_type": "markdown", "metadata": { @@ -728,4 +815,4 @@ }, "nbformat": 4, "nbformat_minor": 1 -} +} \ No newline at end of file