`_, and find the latest Linux installer.
-- Right click the installer, and select **"Copy Link Address"**
-- Head back to your WSL terminal, and type "wget " and then right click next to it. This should paste the link you copied, which should produce something like::
-
- wget https://repo.anaconda.com/archive/Anaconda3-2022.05-Linux-x86_64.sh
-
-- You got it! Not only did you get it, you made it look **easy.** Now just hit enter.
-- At this point, Conda will start installing. Type "yes" and hit Enter for all the various prompts that follow (Accepting the Terms and Conditions, Running Conda Init, etc)
-- Once this is done, close and restart your WSL terminal.
-- Once restarted, verify that conda is working using the following command::
-
- conda env list
-
-Wait wait wait wait just a second.
-Do you realize what just happened?
-
-You've just successfully installed Anaconda!! Hooray!
-Trust me, your life is about to become a LOT easier.
-
-
-- Let's now tap into your newfound powers with Anaconda and create a new virtual environment called "syft_env" by running the following in your WSL shell::
-
- conda create -n syft_env python=3.9 --y
-
-- Let's verify that we created our "syft_env" successfully with the following command (Deja Vu, anyone?)::
-
- conda env list
-
-- You should see two environments in the output. Hooray! Now let's activate the syft virtual env, and let the fun *really* begin::
-
- conda activate syft_env
-
-- Now let's use it to conveniently install a few packages::
-
- sudo apt install python3-pip
- pip3 install pandas matplotlib numpy
- pip3 install jupyterlab
-
-- If the last command fails, try the following instead::
-
- conda install -c conda-forge jupyterlab
-
-
-Step 5: Become the Docker Doctor
-================================
-
-The last tool needed to complete your arsenal is called Docker.
-You can install it by following the instructions `here `_.
-
-Note: The windows user account that launches wsl 2 has to be added to the local group "docker-users". On Windows 10 Home, run netplwiz to add the Windows user to the group "docker-users".
-
-Once you have it running, you just have to ensure the following:
-- You've allocated a sufficient amount of RAM (we recommend atleast 8GB, but you can get by with less)
-- You're using the WSL2 backend
-
-Congratulations, you have reached the end of your journey. Now it is time for your **ultimate test!** Deploying a domain node.
-
-Note that your ultimate test is **optional**- you can do this part later.
-
-
-Step 6: Install Hagrid and PySyft
-=================================
-
-- With the power of WSL and Anaconda, installing our software is as easy as::
-
- pip3 install syft
- pip3 install hagrid
-
-
-Optional: Deploy a Domain Node!
-===============================
-
-Everything we've done so far has been to make this next part as easy as possible. This is the moment we've all been waiting for.
-
-To launch a domain node called "test_domain", ensure your Virtual Environment ("syft_env" in the steps above) is active, that Docker Desktop is running, and run the command below on your WSL terminal::
-
- hagrid launch test_domain
-
-Note: If you get the error message "test_domain is not valid for node_type please use one of the following options: ['domain', 'network']" then rerun the command by changing test_domain to domain.
-
-You should see the containers begin to appear on Docker!
-
-**CONGRATULATIONS!!!**
-
-You have reached the promise land. You're ready to begin remote data science.
-It was a pleasure walking you through the installation process. Now be sure to use your newfound powers and abilities for good!
diff --git a/notebooks/tutorials/deployments/03-deploy-k8s-k3d.ipynb b/notebooks/tutorials/deployments/03-deploy-k8s-k3d.ipynb
index 4c890d6d019..3ba644ee5dc 100644
--- a/notebooks/tutorials/deployments/03-deploy-k8s-k3d.ipynb
+++ b/notebooks/tutorials/deployments/03-deploy-k8s-k3d.ipynb
@@ -70,7 +70,28 @@
"Install PySyft on the Kubernetes cluster with your preferred version:\n",
"```sh\n",
"helm install my-syft openmined/syft --version $SYFT_VERSION --namespace syft --create-namespace --set ingress.className=\"traefik\"\n",
- "```"
+ "```\n",
+ "\n",
+ "\n",
+ "💡 Tip 1:\n",
+ "\n",
+ "If you want to deploy your Kubernetes cluster in a resource-constrained environment, use the following flags to override the default configurations. Please note that you will need at least 1 CPU and 2 GB of RAM on Docker, and some tests may not work in such low-resource environments:\n",
+ "\n",
+ "```sh\n",
+ "helm install my-syft openmined/syft --version $SYFT_VERSION --namespace syft --create-namespace --set ingress.className=\"traefik\" --set node.resourcesPreset=null --set seaweedfs.resourcesPreset=null --set mongo.resourcesPreset=null --set registry.resourcesPreset=null --set proxy.resourcesPreset=null --set frontend.resourcesPreset=null\n",
+ "```\n",
+ "\n",
+ "
\n",
+ "\n",
+ "\n",
+ "💡 Tip 2:\n",
+ "\n",
+ "If you would like to set your own default password even for the production style deployment, use the following command:\n",
+ "\n",
+ "```sh\n",
+ "helm install my-syft openmined/syft --version $SYFT_VERSION --namespace syft --create-namespace --set ingress.className=\"traefik\" --set global.randomizedSecrets=false --set node.secret.defaultRootPassword=\"changethis\" --set seaweedfs.secret.s3RootPassword=\"admin\" --set mongo.secret.rootPassword=\"example\"\n",
+ "```\n",
+ "
\n"
]
},
{
@@ -90,7 +111,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "```python3\n",
+ "```python\n",
"# syft absolute\n",
"import syft as sy\n",
"\n",
@@ -103,6 +124,7 @@
"metadata": {},
"source": [
"This will return a node handle by connecting to `http://localhost:8080` which is the default host and port where your kubernetes cluster will be running. You can connect to a different host and port by setting the environment variables `NODE_URL` and `NODE_PORT`.\n",
+ "\n",
"```python\n",
"import os\n",
"\n",
@@ -115,14 +137,25 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Now, we are ready to start using the domain. The domain comes with default login credentials for the admin."
+ "Now, we are ready to start using the domain. Since helm is a product grade deployement stack, the domain comes with a randomized password for the default email credentials for the admin. Either run with Step 5 with your custom password or to extract the randomized password using `kubectl`, run the following command (in case you use a custom cluster name in step 1, replace `--context=k3d-$CLUSTER_NAME` appropriately): \n",
+ "\n",
+ "```sh\n",
+ "kubectl --context=k3d-syft get secret backend-secret -n syft -o jsonpath='{.data.defaultRootPassword}' | base64 --decode\n",
+ "```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "```python3\n",
+ "and use the password instead of \"changethis\" below:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "```python\n",
"client = node.login(email=\"info@openmined.org\", password=\"changethis\")\n",
"```"
]
diff --git a/notebooks/tutorials/deployments/07-deploy-devspace.ipynb b/notebooks/tutorials/deployments/07-deploy-devspace.ipynb
index 71c158afddd..29632c99dec 100644
--- a/notebooks/tutorials/deployments/07-deploy-devspace.ipynb
+++ b/notebooks/tutorials/deployments/07-deploy-devspace.ipynb
@@ -4,7 +4,126 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "TODO"
+ "# Developing over PySyft\n",
+ "This guide is meant for developers trying to deploy PySyft locally on a Kubernetes cluster. PySyft is a powerful framework for privacy-preserving data analysis and machine learning, and deploying it on Kubernetes allows an easy way to quickly try out the full PySyft stack on your own system. This guide will walk you through the process step by step. \n",
+ "\n",
+ "Follow [00-deployment-types.ipynb](./00-deployment-types.ipynb) for other deployment options and an overview of different deployment types."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "vscode": {
+ "languageId": "plaintext"
+ }
+ },
+ "source": [
+ "## Prerequisites\n",
+ "Before we begin, ensure you have the following prerequisites installed on your system. You may need a package management systems like [brew](https://brew.sh/) for installation of dependencies and packages. \n",
+ "* [devspace](https://www.devspace.sh/docs/getting-started/installation?x0=3): DevSpace lets you automate all build and deployment steps and enables interactive modification.\n",
+ "* [Docker](https://docs.docker.com/install/): Docker is required to create and manage containers.\n",
+ "* [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl): kubectl is the command-line tool for interacting with Kubernetes clusters.\n",
+ "* [k3d](https://k3d.io/v5.6.3/#installation): k3d is used to create local Kubernetes clusters.\n",
+ "* [tox](https://pypi.org/project/tox/): We use tox as a command executor. \n",
+ "* [uv](): This allows for faster tox environment builds and speeds up dev time. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "vscode": {
+ "languageId": "plaintext"
+ }
+ },
+ "source": [
+ "## Dev Environment set-up"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "vscode": {
+ "languageId": "plaintext"
+ }
+ },
+ "source": [
+ "In order to perform an isolated setup, create a virtual environment. Below are some instructions to get started (example `environment.yml` and `requirements.txt` file are [provided below](#dependency-files)):\n",
+ "| Conda | Venv |\n",
+ "|:---|:---|\n",
+ "| 1. Create `environment.yml` with name `syft-dev`
2. Run `conda env create -f environment.yml`
3. Run `conda activate syft-dev` | 1. Create a `requirements.txt` file
2. Run `python -m venv syft-dev`
3. Run `.\\syft-dev\\Scripts\\activate` for Windows or
`source syft-dev/bin/activate` for Mac/Linux
4. `pip install -r requirements.txt`
5. `pip install -e \"packages/syft[ dev ]\"` |\n",
+ "\n",
+ "If you would like to do a non-isolated install then install the dependencies using [brew](https://brew.sh/) as follows: \n",
+ "* `brew install helm devspace k9s k3d`\n",
+ "* `pip install -e 'packages/syft[dev]'`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "vscode": {
+ "languageId": "plaintext"
+ }
+ },
+ "source": [
+ "## Running the k8s cluster\n",
+ "\n",
+ "We have a number of tox commands that help with the setup of the local k8s cluster.\n",
+ "\n",
+ "* For starting the cluster & registry (run this once initially or after every hard reset)\n",
+ " - `tox -e dev.k8s.start` to start a local registry + cluster and patch hosts file\n",
+ "* For deploying syft (run this each time you update the code or helm charts)\n",
+ " - `tox -e dev.k8s.deploy` to deploy syft on the cluster\n",
+ " - `tox -e dev.k8s.hotreload` to deploy syft on the cluster with hot reloading and port forwards (run once and keep editing that code)\n",
+ "* For cleaning up (run as required)\n",
+ " - `tox -e dev.k8s.cleanup` only removes the syft deployment but keep the cluster running (soft reset, re-run using `dev.k8s.deploy`)\n",
+ " - `tox -e dev.k8s.destroy` deletes the syft deployment, cluster, but keeps the registry running (mid reset, re-run using `dev.k8s.start`)\n",
+ " - `tox -e dev.k8s.destroyall` deletes the syft deployment, cluster and registry (hard reset, run re-run using `dev.k8s.start`)\n",
+ "\n",
+ "You can use the command `k9s` or `lazydocker` (will require installing lazydocker) to view your cluster.\n",
+ "\n",
+ "\n",
+ "#### Launching Different Node Types\n",
+ "Alternatively, you can also use the following commands to launch different node types:\n",
+ "* Launching a Domain: `CLUSTER_NAME=testdomain1 CLUSTER_HTTP_PORT=9082 tox -e dev.k8s.launch.domain`\n",
+ "* Launching a Gateway: `CLUSTER_NAME=testgateway1 CLUSTER_HTTP_PORT=9081 tox -e dev.k8s.launch.gateway`\n",
+ "* Launching a Enclave: `CLUSTER_NAME=testenclave1 CLUSTER_HTTP_PORT=9083 tox -e dev.k8s.launch.enclave`\n",
+ "* Launching Nodes with `hotreload` using tox posargs: `tox -e dev.k8s.launch.domain -- hotreload`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Next Steps\n",
+ "Congratulations! You have successfully deployed PySyft on your local Kubernetes cluster. Now, you can explore its capabilities and use cases through our API example notebooks:\n",
+ "\n",
+ "📝 [API Example Notebooks](../../api)\n",
+ "- [00-load-data.ipynb](../../api/0.8/00-load-data.ipynb)\n",
+ "- [01-submit-code.ipynb](../../api/0.8/01-submit-code.ipynb)\n",
+ "- [02-review-code-and-approve.ipynb](../../api/0.8/02-review-code-and-approve.ipynb)\n",
+ "- [03-data-scientist-download-result.ipynb](../../api/0.8/03-data-scientist-download-result.ipynb)\n",
+ "- [04-jax-example.ipynb](../../api/0.8/04-jax-example.ipynb)\n",
+ "- [05-custom-policy.ipynb](../../api/0.8/05-custom-policy.ipynb)\n",
+ "- [06-multiple-code-requests.ipynb](../../api/0.8/06-multiple-code-requests.ipynb)\n",
+ "- [07-domain-register-control-flow.ipynb](../../api/0.8/07-domain-register-control-flow.ipynb)\n",
+ "- [08-code-version.ipynb](../../api/0.8/08-code-version.ipynb)\n",
+ "- [09-blob-storage.ipynb](../../api/0.8/09-blob-storage.ipynb)\n",
+ "- [10-container-images.ipynb](../../api/0.8/10-container-images.ipynb)\n",
+ "- [11-container-images-k8s.ipynb](../../api/0.8/11-container-images-k8s.ipynb)\n",
+ "\n",
+ "Feel free to explore these notebooks to get started with PySyft and unlock its full potential for privacy-preserving machine learning!"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Dependency files\n",
+ "Example `environment.yml` and `requirement.txt` files:\n",
+ "\n",
+ "| Conda | Venv |\n",
+ "|:---|:---|\n",
+ "| `environment.yml`
name: syft-dev
channels:
- conda-forge
dependencies:
- python=3.12
- jupyter
- ipython
- ipykernel
- pandas
- pre-commit
- black
- mypy
- flake8
- ruff
- isort
- pip
- pip:
- pipdeptree
- johnnydep
- tox
- tox-uv
- uvloop
- \"-e packages/syft[dev]\" | `requirements.txt`
# Standard dependencies
jupyter
ipython
ipykernel
pandas
pre-commit
black
mypy
flake8
ruff
isort
# Pip dependencies
pipdeptree
johnnydep
tox
tox-uv
uvloop |\n"
]
}
],
diff --git a/notebooks/tutorials/hello-syft/01-hello-syft.ipynb b/notebooks/tutorials/hello-syft/01-hello-syft.ipynb
index 8a7f6a674d2..12f01679e3c 100644
--- a/notebooks/tutorials/hello-syft/01-hello-syft.ipynb
+++ b/notebooks/tutorials/hello-syft/01-hello-syft.ipynb
@@ -83,7 +83,7 @@
"source": [
"## Launch a dummy server \n",
"\n",
- "In this tutorial, for the sake of demonstration, we will be using in-memory workers as dummy servers. For details of deploying a server on your own using `syft` and `hagrid`, please refer to the `quickstart` tutorials."
+ "In this tutorial, for the sake of demonstration, we will be using in-memory workers as dummy servers. For details of deploying a server on your own using `syft`."
]
},
{
diff --git a/packages/grid/frontend/src/lib/components/Datasets/DatasetModalNew.svelte b/packages/grid/frontend/src/lib/components/Datasets/DatasetModalNew.svelte
index 66587a5fd63..3b0cb81318a 100644
--- a/packages/grid/frontend/src/lib/components/Datasets/DatasetModalNew.svelte
+++ b/packages/grid/frontend/src/lib/components/Datasets/DatasetModalNew.svelte
@@ -43,10 +43,10 @@
>
2
- Install HAGrid by running the code below in your Jupyter Notebook
+
-
pip install -U hagrid
+
@@ -57,12 +57,10 @@
3
- Once HAGrid is installed open the "Upload Dataset" quickstart tutorial notebook by
- running the code below in your Jupyter Notebook.
diff --git a/packages/grid/helm/syft/templates/global/ingress.yaml b/packages/grid/helm/syft/templates/global/ingress.yaml
index 677a66313a6..a76c48d8260 100644
--- a/packages/grid/helm/syft/templates/global/ingress.yaml
+++ b/packages/grid/helm/syft/templates/global/ingress.yaml
@@ -20,13 +20,18 @@ spec:
{{- end }}
defaultBackend:
service:
+ {{- if .Values.proxy.enabled }}
name: proxy
+ {{- else }}
+ name: frontend
+ {{- end }}
port:
number: 80
rules:
- host: {{ .Values.ingress.hostname | quote }}
http:
paths:
+ {{- if .Values.proxy.enabled }}
- backend:
service:
name: proxy
@@ -34,6 +39,36 @@ spec:
number: 80
path: /
pathType: Prefix
+ {{- else }}
+ - backend:
+ service:
+ name: frontend
+ port:
+ number: 80
+ path: /
+ pathType: Prefix
+ - backend:
+ service:
+ name: backend
+ port:
+ number: 80
+ path: /()(api)(.*)
+ pathType: Prefix
+ - backend:
+ service:
+ name: backend
+ port:
+ number: 80
+ path: /()(docs|redoc)
+ pathType: Prefix
+ - backend:
+ service:
+ name: seaweedfs
+ port:
+ number: 8333
+ path: /(blob/)(.*)
+ pathType: ImplementationSpecific
+ {{- end }}
{{- if .Values.ingress.tls.enabled }}
tls:
- hosts:
diff --git a/packages/grid/helm/syft/templates/proxy/proxy-configmap.yaml b/packages/grid/helm/syft/templates/proxy/proxy-configmap.yaml
index 1989f399161..2ebc6d6a9ba 100644
--- a/packages/grid/helm/syft/templates/proxy/proxy-configmap.yaml
+++ b/packages/grid/helm/syft/templates/proxy/proxy-configmap.yaml
@@ -1,3 +1,4 @@
+{{- if .Values.proxy.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
@@ -72,4 +73,4 @@ data:
providers:
file:
filename: /etc/traefik/dynamic.yml
-
+{{- end }}
diff --git a/packages/grid/helm/syft/templates/proxy/proxy-deployment.yaml b/packages/grid/helm/syft/templates/proxy/proxy-deployment.yaml
index 6adb42f6c9c..69b60f905ab 100644
--- a/packages/grid/helm/syft/templates/proxy/proxy-deployment.yaml
+++ b/packages/grid/helm/syft/templates/proxy/proxy-deployment.yaml
@@ -1,3 +1,4 @@
+{{- if .Values.proxy.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -59,3 +60,4 @@ spec:
- configMap:
name: proxy-config
name: traefik-conf
+{{- end }}
diff --git a/packages/grid/helm/syft/templates/proxy/proxy-service.yaml b/packages/grid/helm/syft/templates/proxy/proxy-service.yaml
index 464f4338d33..72a9a80c3bf 100644
--- a/packages/grid/helm/syft/templates/proxy/proxy-service.yaml
+++ b/packages/grid/helm/syft/templates/proxy/proxy-service.yaml
@@ -1,3 +1,4 @@
+{{- if .Values.proxy.enabled }}
apiVersion: v1
kind: Service
metadata:
@@ -18,3 +19,4 @@ spec:
protocol: TCP
port: 80
targetPort: 80
+{{- end }}
diff --git a/packages/grid/helm/syft/templates/registry/registry-service.yaml b/packages/grid/helm/syft/templates/registry/registry-service.yaml
index c132545bf2c..89386a626cf 100644
--- a/packages/grid/helm/syft/templates/registry/registry-service.yaml
+++ b/packages/grid/helm/syft/templates/registry/registry-service.yaml
@@ -1,3 +1,4 @@
+{{- if .Values.registry.enabled }}
apiVersion: v1
kind: Service
metadata:
@@ -15,3 +16,4 @@ spec:
protocol: TCP
port: 80
targetPort: 5000
+{{- end }}
diff --git a/packages/grid/helm/syft/templates/registry/registry-statefulset.yaml b/packages/grid/helm/syft/templates/registry/registry-statefulset.yaml
index baefc45dce0..028ef6d8dad 100644
--- a/packages/grid/helm/syft/templates/registry/registry-statefulset.yaml
+++ b/packages/grid/helm/syft/templates/registry/registry-statefulset.yaml
@@ -1,3 +1,4 @@
+{{- if .Values.registry.enabled }}
apiVersion: apps/v1
kind: StatefulSet
metadata:
@@ -73,3 +74,4 @@ spec:
resources:
requests:
storage: {{ .Values.registry.storageSize | quote }}
+{{- end }}
diff --git a/packages/grid/helm/syft/values.yaml b/packages/grid/helm/syft/values.yaml
index 69ac1413291..e44d2cf7982 100644
--- a/packages/grid/helm/syft/values.yaml
+++ b/packages/grid/helm/syft/values.yaml
@@ -112,6 +112,7 @@ seaweedfs:
# =================================================================================
proxy:
+ enabled: true
# Extra environment vars
env: null
@@ -132,6 +133,7 @@ proxy:
# =================================================================================
registry:
+ enabled: true
# Extra environment vars
env: null
@@ -201,6 +203,13 @@ node:
ingress:
hostname: null # do not make this localhost
+ annotations: null
+
+ # When using nginx ingress controller without internal proxy , re-enable this
+ # annotations:
+ # nginx.ingress.kubernetes.io/use-regex: "true"
+ # nginx.ingress.kubernetes.io/rewrite-target: /$2$3
+ # nginx.ingress.kubernetes.io/upstream-vhost: "$service_name:$service_port"
tls:
enabled: false
@@ -236,4 +245,4 @@ attestation:
# =================================================================================
-extraResources: []
\ No newline at end of file
+extraResources: []
diff --git a/packages/syft/setup.cfg b/packages/syft/setup.cfg
index fb43665b787..ac91f01c061 100644
--- a/packages/syft/setup.cfg
+++ b/packages/syft/setup.cfg
@@ -48,6 +48,7 @@ syft =
typing_extensions==4.10.0
sherlock[filelock]==0.4.1
uvicorn[standard]==0.27.1
+ markdown==3.5.2
fastapi==0.110.0
psutil==5.9.8
itables==1.7.1
diff --git a/packages/syft/src/syft/client/client.py b/packages/syft/src/syft/client/client.py
index ba4dfc38c80..3e27b445015 100644
--- a/packages/syft/src/syft/client/client.py
+++ b/packages/syft/src/syft/client/client.py
@@ -730,7 +730,7 @@ def numpy(self) -> APIModule | None:
@property
def settings(self) -> APIModule | None:
- if self.api.has_service("user"):
+ if self.api.has_service("settings"):
return self.api.services.settings
return None
diff --git a/packages/syft/src/syft/client/domain_client.py b/packages/syft/src/syft/client/domain_client.py
index 8f1e7cb9dc8..ac85e965e78 100644
--- a/packages/syft/src/syft/client/domain_client.py
+++ b/packages/syft/src/syft/client/domain_client.py
@@ -4,11 +4,14 @@
# stdlib
from pathlib import Path
import re
+from string import Template
from typing import TYPE_CHECKING
from typing import cast
# third party
from loguru import logger
+import markdown
+from result import Result
from tqdm import tqdm
# relative
@@ -26,11 +29,9 @@
from ..service.sync.sync_state import SyncState
from ..service.user.roles import Roles
from ..service.user.user import UserView
-from ..service.user.user_roles import ServiceRole
from ..types.blob_storage import BlobFile
from ..types.uid import UID
-from ..util.assets import load_png_base64
-from ..util.notebook_ui.styles import FONT_CSS
+from ..util.misc_objs import HTMLObject
from ..util.util import get_mb_size
from ..util.util import prompt_warning_message
from .api import APIModule
@@ -325,6 +326,11 @@ def _get_service_by_name_if_exists(self, name: str) -> APIModule | None:
return getattr(self.api.services, name)
return None
+ def set_node_side_type_dangerous(
+ self, node_side_type: str
+ ) -> Result[SyftSuccess, SyftError]:
+ return self.api.services.settings.set_node_side_type_dangerous(node_side_type)
+
@property
def data_subject_registry(self) -> APIModule | None:
return self._get_service_by_name_if_exists("data_subject")
@@ -404,107 +410,15 @@ def get_project(
return self.api.services.project.get_all()
def _repr_html_(self) -> str:
- guest_commands = """
- <your_client>.datasets - list datasets
- <your_client>.code - list code
- <your_client>.login - list projects
-
- <your_client>.code.submit? - display function signature
- """
- ds_commands = """
- <your_client>.datasets - list datasets
- <your_client>.code - list code
- <your_client>.projects - list projects
-
- <your_client>.code.submit? - display function signature
- """
-
- do_commands = """
- <your_client>.projects - list projects
- <your_client>.requests - list requests
- <your_client>.users - list users
-
- <your_client>.requests.submit? - display function signature
- """
-
- # TODO: how to select ds/do commands based on self.__user_role
-
- if (
- self.user_role.value == ServiceRole.NONE.value
- or self.user_role.value == ServiceRole.GUEST.value
- ):
- commands = guest_commands
- elif (
- self.user_role is not None
- and self.user_role.value == ServiceRole.DATA_SCIENTIST.value
- ):
- commands = ds_commands
- elif (
- self.user_role is not None
- and self.user_role.value >= ServiceRole.DATA_OWNER.value
- ):
- commands = do_commands
-
- command_list = f"""
-
- """
-
- small_grid_symbol_logo = load_png_base64("small-grid-symbol-logo.png")
-
- url = getattr(self.connection, "url", None)
- node_details = f"URL: {url}
" if url else ""
- if self.metadata is not None:
- node_details += f"Node Type: {self.metadata.node_type.capitalize()}
"
- node_side_type = (
- "Low Side"
- if self.metadata.node_side_type == NodeSideType.LOW_SIDE.value
- else "High Side"
- )
- node_details += f"Node Side Type: {node_side_type}
"
- node_details += (
- f"Syft Version: {self.metadata.syft_version}
"
- )
+ obj = self.api.services.settings.welcome_show()
+ if isinstance(obj, SyftError):
+ return obj.message
+ updated_template_str = Template(obj.text).safe_substitute(
+ node_url=getattr(self.connection, "url", None)
+ )
+ # If it's a markdown structured file
+ if not isinstance(obj, HTMLObject):
+ return markdown.markdown(updated_template_str)
- self._fetch_node_metadata(self.credentials)
- return f"""
-
-
-
-
Welcome to {self.name}
-
- {node_details}
-
-
-
Commands to Get Started
- {command_list}
-
- """
+ # if it's a html string
+ return updated_template_str
diff --git a/packages/syft/src/syft/node/node.py b/packages/syft/src/syft/node/node.py
index 64d10f92930..2340ca33ba2 100644
--- a/packages/syft/src/syft/node/node.py
+++ b/packages/syft/src/syft/node/node.py
@@ -93,6 +93,7 @@
from ..service.service import ServiceConfigRegistry
from ..service.service import UserServiceConfigRegistry
from ..service.settings.settings import NodeSettings
+from ..service.settings.settings import NodeSettingsUpdate
from ..service.settings.settings_service import SettingsService
from ..service.settings.settings_stash import SettingsStash
from ..service.sync.sync_service import SyncService
@@ -120,6 +121,8 @@
from ..store.mongo_document_store import MongoStoreConfig
from ..store.sqlite_document_store import SQLiteStoreClientConfig
from ..store.sqlite_document_store import SQLiteStoreConfig
+from ..types.syft_metaclass import Empty
+from ..types.syft_object import PartialSyftObject
from ..types.syft_object import SYFT_OBJECT_VERSION_2
from ..types.syft_object import SyftObject
from ..types.uid import UID
@@ -994,6 +997,16 @@ def remove_temp_dir(self) -> None:
if rootdir.exists():
shutil.rmtree(rootdir, ignore_errors=True)
+ def update_self(self, settings: NodeSettings) -> None:
+ updateable_attrs = (
+ NodeSettingsUpdate.model_fields.keys()
+ - PartialSyftObject.model_fields.keys()
+ )
+ for attr_name in updateable_attrs:
+ attr = getattr(settings, attr_name)
+ if attr is not Empty:
+ setattr(self, attr_name, attr)
+
@property
def settings(self) -> NodeSettings:
settings_stash = SettingsStash(store=self.document_store)
@@ -1006,21 +1019,20 @@ def settings(self) -> NodeSettings:
)
if settings.is_ok() and len(settings.ok()) > 0:
settings = settings.ok()[0]
+ self.update_self(settings)
return settings
@property
def metadata(self) -> NodeMetadataV3:
- name = ""
- organization = ""
- description = ""
- show_warnings = self.enable_warnings
settings_data = self.settings
name = settings_data.name
organization = settings_data.organization
description = settings_data.description
show_warnings = settings_data.show_warnings
- node_type = self.node_type.value if self.node_type else ""
- node_side_type = self.node_side_type.value if self.node_side_type else ""
+ node_type = settings_data.node_type.value if settings_data.node_type else ""
+ node_side_type = (
+ settings_data.node_side_type.value if settings_data.node_side_type else ""
+ )
return NodeMetadataV3(
name=name,
diff --git a/packages/syft/src/syft/protocol/protocol_version.json b/packages/syft/src/syft/protocol/protocol_version.json
index b4050ab030f..36f59d41b20 100644
--- a/packages/syft/src/syft/protocol/protocol_version.json
+++ b/packages/syft/src/syft/protocol/protocol_version.json
@@ -18,6 +18,11 @@
"version": 3,
"hash": "0f812fdd5aecc3e3aa1a7c953bbf7f8d8b03a77c5cdbb37e981fa91c8134c9f4",
"action": "add"
+ },
+ "4": {
+ "version": 4,
+ "hash": "ec783a7cd097e2bc4273a519d11023c796aebb9e3710c1d8332c0e46966d4ae0",
+ "action": "add"
}
},
"NodeSettings": {
@@ -25,6 +30,11 @@
"version": 4,
"hash": "318e578f8a9af213a6af0cc2c567b62196b0ff81769d808afff4dd1eb7c372b8",
"action": "add"
+ },
+ "5": {
+ "version": 5,
+ "hash": "cde18eb23fdffcfba47bc0e85efdbba1d59f1f5d6baa9c9690e1af14b35eb74e",
+ "action": "add"
}
},
"EnclaveMetadata": {
@@ -198,6 +208,13 @@
"hash": "e5f099940a7623f145f51f3e15b97a910a1d7fda1f67739420fed3035d1f2995",
"action": "add"
}
+ },
+ "HTMLObject": {
+ "1": {
+ "version": 1,
+ "hash": "010d9aaca95f3fdfc8d1f97d01c1bd66483da774a59275b310c08d6912f7f863",
+ "action": "add"
+ }
}
}
}
diff --git a/packages/syft/src/syft/service/api/api.py b/packages/syft/src/syft/service/api/api.py
index 0b55a0746a4..6977f29e8c7 100644
--- a/packages/syft/src/syft/service/api/api.py
+++ b/packages/syft/src/syft/service/api/api.py
@@ -31,8 +31,8 @@
from ...types.transforms import keep
from ...types.transforms import transform
from ...types.uid import UID
+from ...util.misc_objs import MarkdownDescription
from ..context import AuthedServiceContext
-from ..dataset.dataset import MarkdownDescription
from ..response import SyftError
from ..user.user import UserView
diff --git a/packages/syft/src/syft/service/dataset/dataset.py b/packages/syft/src/syft/service/dataset/dataset.py
index 401ae63a44f..687a2eb5f84 100644
--- a/packages/syft/src/syft/service/dataset/dataset.py
+++ b/packages/syft/src/syft/service/dataset/dataset.py
@@ -6,7 +6,6 @@
from typing import Any
# third party
-from IPython.display import HTML
from IPython.display import display
import itables
import pandas as pd
@@ -35,6 +34,7 @@
from ...util.colors import SURFACE
from ...util.colors import SURFACE_SURFACE
from ...util.markdown import as_markdown_python_code
+from ...util.misc_objs import MarkdownDescription
from ...util.notebook_ui.icons import Icon
from ...util.notebook_ui.styles import FONT_CSS
from ...util.notebook_ui.styles import ITABLES_CSS
@@ -86,31 +86,6 @@ def __hash__(self) -> int:
return hash(self.email)
-@serializable()
-class MarkdownDescription(SyftObject):
- # version
- __canonical_name__ = "MarkdownDescription"
- __version__ = SYFT_OBJECT_VERSION_2
-
- text: str
-
- def _repr_markdown_(self, wrap_as_python: bool = True, indent: int = 0) -> str:
- style = """
-
- """
- display(HTML(style))
- return self.text
-
-
@serializable()
class Asset(SyftObject):
# version
diff --git a/packages/syft/src/syft/service/settings/settings.py b/packages/syft/src/syft/service/settings/settings.py
index aa4d5d6719c..0ac31b9ff7d 100644
--- a/packages/syft/src/syft/service/settings/settings.py
+++ b/packages/syft/src/syft/service/settings/settings.py
@@ -2,12 +2,17 @@
from collections.abc import Callable
from typing import Any
+# third party
+from IPython.display import display
+from pydantic import field_validator
+
# relative
from ...abstract_node import NodeSideType
from ...abstract_node import NodeType
from ...node.credentials import SyftVerifyKey
from ...serde.serializable import serializable
from ...service.worker.utils import DEFAULT_WORKER_POOL_NAME
+from ...types.syft_metaclass import Empty
from ...types.syft_migration import migrate
from ...types.syft_object import PartialSyftObject
from ...types.syft_object import SYFT_OBJECT_VERSION_2
@@ -19,6 +24,10 @@
from ...types.uid import UID
from ...util import options
from ...util.colors import SURFACE
+from ...util.misc_objs import HTMLObject
+from ...util.misc_objs import MarkdownDescription
+from ...util.schema import DEFAULT_WELCOME_MSG
+from ..response import SyftInfo
@serializable()
@@ -36,7 +45,7 @@ class NodeSettingsUpdateV2(PartialSyftObject):
@serializable()
-class NodeSettingsUpdate(PartialSyftObject):
+class NodeSettingsUpdateV3(PartialSyftObject):
__canonical_name__ = "NodeSettingsUpdate"
__version__ = SYFT_OBJECT_VERSION_3
@@ -50,8 +59,85 @@ class NodeSettingsUpdate(PartialSyftObject):
association_request_auto_approval: bool
+@serializable()
+class NodeSettingsUpdate(PartialSyftObject):
+ __canonical_name__ = "NodeSettingsUpdate"
+ __version__ = SYFT_OBJECT_VERSION_4
+ id: UID
+ name: str
+ organization: str
+ description: str
+ on_board: bool
+ signup_enabled: bool
+ admin_email: str
+ association_request_auto_approval: bool
+ welcome_markdown: HTMLObject | MarkdownDescription
+ node_side_type: str
+
+ @field_validator("node_side_type", check_fields=False)
+ @classmethod
+ def validate_node_side_type(cls, v: str) -> type[Empty]:
+ msg = f"You cannot update 'node_side_type' through NodeSettingsUpdate. \
+Please use client.set_node_side_type_dangerous(node_side_type={v}). \
+Be aware if you have private data on the node and you want to change it to the Low Side, \
+as information might be leaked."
+ try:
+ display(SyftInfo(message=msg))
+ except Exception:
+ print(SyftInfo(message=msg))
+ return Empty
+
+
@serializable()
class NodeSettings(SyftObject):
+ __canonical_name__ = "NodeSettings"
+ __version__ = 5
+ __repr_attrs__ = [
+ "name",
+ "organization",
+ "deployed_on",
+ "signup_enabled",
+ "admin_email",
+ ]
+
+ id: UID
+ name: str = "Node"
+ deployed_on: str
+ organization: str = "OpenMined"
+ verify_key: SyftVerifyKey
+ on_board: bool = True
+ description: str = "Text"
+ node_type: NodeType = NodeType.DOMAIN
+ signup_enabled: bool
+ admin_email: str
+ node_side_type: NodeSideType = NodeSideType.HIGH_SIDE
+ show_warnings: bool
+ association_request_auto_approval: bool
+ default_worker_pool: str = DEFAULT_WORKER_POOL_NAME
+ welcome_markdown: HTMLObject | MarkdownDescription = HTMLObject(
+ text=DEFAULT_WELCOME_MSG
+ )
+
+ def _repr_html_(self) -> Any:
+ return f"""
+
+
+
Settings
+
Id: {self.id}
+
Name: {self.name}
+
Organization: {self.organization}
+
Deployed on: {self.deployed_on}
+
Signup enabled: {self.signup_enabled}
+
Admin email: {self.admin_email}
+
+
+
Welcome to $domain_name
+
+ URL: $node_url
+ Node Type: $node_type
+ Node Side Type:$node_side_type
+ Syft Version: $node_version
+
+
+
+
Commands to Get Started
+ $command_list
+
+ """
+
# json schema primitive types
primitive_mapping = {
list: "array",
diff --git a/tox.ini b/tox.ini
index 35cb08c9fb7..c374b12ff36 100644
--- a/tox.ini
+++ b/tox.ini
@@ -631,6 +631,51 @@ commands =
bash -c 'cd packages/grid/helm/repo && \
helm repo index . --url https://openmined.github.io/PySyft/helm'
+
+[testenv:dev.k8s.ready]
+description = Check readiness of k8s deployement
+changedir = {toxinidir}/packages/grid
+allowlist_externals =
+ bash
+ tox
+ curl
+setenv =
+ CLUSTER_NAME = {env:CLUSTER_NAME:syft}
+ CLUSTER_HTTP_PORT = {env:NODE_PORT:8080}
+; Usage for posargs: names of the relevant services among {frontend backend proxy mongo seaweedfs registry}
+commands =
+ bash -c "env; date; k3d version"
+
+ # Frontend
+ bash -c "if echo '{posargs}' | grep -q 'frontend'; then \
+ echo 'Checking readiness of frontend'; \
+ ./scripts/wait_for.sh service frontend --context k3d-$CLUSTER_NAME --namespace syft && \
+ (kubectl logs service/frontend --context k3d-$CLUSTER_NAME --namespace syft -f &) | grep -q -E 'Network:\s+https?://[a-zA-Z0-9.-]+:[0-9]+/' || true; \
+ fi"
+
+ # Backend
+ bash -c "if echo '{posargs}' | grep -q 'backend'; then \
+ echo 'Checking readiness of backend'; \
+ ./scripts/wait_for.sh service backend --context k3d-$CLUSTER_NAME --namespace syft && \
+ (kubectl logs service/backend --context k3d-$CLUSTER_NAME --namespace syft -f &) | grep -q 'Application startup complete' || true; \
+ fi"
+
+ # Mongo
+ bash -c "if echo '{posargs}' | grep -q 'mongo'; then echo 'Checking readiness of Mongo'; ./scripts/wait_for.sh service mongo --context k3d-$CLUSTER_NAME --namespace syft; fi"
+
+ # Proxy
+ bash -c "if echo '{posargs}' | grep -q 'proxy'; then echo 'Checking readiness of proxy'; ./scripts/wait_for.sh service proxy --context k3d-$CLUSTER_NAME --namespace syft; fi"
+
+ # Seaweedfs
+ bash -c "if echo '{posargs}' | grep -q 'seaweedfs'; then echo 'Checking readiness of SeaweedFS'; ./scripts/wait_for.sh service seaweedfs --context k3d-$CLUSTER_NAME --namespace syft; fi"
+
+ # Registry
+ bash -c "if echo '{posargs}' | grep -q 'registry'; then echo 'Checking readiness of Registry'; ./scripts/wait_for.sh service registry --context k3d-$CLUSTER_NAME --namespace syft; fi"
+
+ # Extra
+ bash -c "curl http://localhost:${CLUSTER_HTTP_PORT}/api/v2/metadata"
+
+
[testenv:syft.test.helm]
description = Test Helm Chart for Kubernetes
changedir = {toxinidir}/packages/grid
@@ -648,6 +693,7 @@ setenv =
; env vars for dev.k8s.start
CLUSTER_NAME = testdomain
CLUSTER_HTTP_PORT = {env:NODE_PORT:8080}
+; Usage for posargs: if you pass override to this tox command, then resourcesPreset will be overridden
commands =
bash -c "env; date; k3d version"
@@ -657,21 +703,26 @@ commands =
bash -c 'if [[ $SYFT_VERSION == "local" ]]; then \
echo "Installing local helm charts"; \
- helm install ${CLUSTER_NAME} ./helm/syft -f ./helm/values.dev.yaml --kube-context k3d-${CLUSTER_NAME} --namespace syft --create-namespace; \
+ if [[ "{posargs}" == "override" ]]; then \
+ echo "Overriding resourcesPreset"; \
+ helm install ${CLUSTER_NAME} ./helm/syft -f ./helm/values.dev.yaml --kube-context k3d-${CLUSTER_NAME} --namespace syft --create-namespace --set node.resourcesPreset=null --set seaweedfs.resourcesPreset=null --set mongo.resourcesPreset=null --set registry.resourcesPreset=null --set proxy.resourcesPreset=null --set frontend.resourcesPreset=null; \
+ else \
+ helm install ${CLUSTER_NAME} ./helm/syft -f ./helm/values.dev.yaml --kube-context k3d-${CLUSTER_NAME} --namespace syft --create-namespace; \
+ fi \
else \
echo "Installing helm charts from repo for syft version: ${SYFT_VERSION}"; \
helm repo add openmined https://openmined.github.io/PySyft/helm; \
helm repo update openmined; \
- helm install ${CLUSTER_NAME} openmined/syft --version=${SYFT_VERSION} -f ./helm/values.dev.yaml --kube-context k3d-${CLUSTER_NAME} --namespace syft --create-namespace; \
+ if [[ "{posargs}" == "override" ]]; then \
+ echo "Overriding resourcesPreset"; \
+ helm install ${CLUSTER_NAME} openmined/syft --version=${SYFT_VERSION} -f ./helm/values.dev.yaml --kube-context k3d-${CLUSTER_NAME} --namespace syft --create-namespace --set node.resourcesPreset=null --set seaweedfs.resourcesPreset=null --set mongo.resourcesPreset=null --set registry.resourcesPreset=null --set proxy.resourcesPreset=null --set frontend.resourcesPreset=null; \
+ else \
+ helm install ${CLUSTER_NAME} openmined/syft --version=${SYFT_VERSION} -f ./helm/values.dev.yaml --kube-context k3d-${CLUSTER_NAME} --namespace syft --create-namespace; \
+ fi \
fi'
; wait for everything else to be loaded
- bash -c './scripts/wait_for.sh service frontend --context k3d-$CLUSTER_NAME --namespace syft'
- bash -c '(kubectl logs service/frontend --context k3d-$CLUSTER_NAME --namespace syft -f &) | grep -q -E "Network:\s+https?://[a-zA-Z0-9.-]+:[0-9]+/" || true'
- bash -c './scripts/wait_for.sh service mongo --context k3d-$CLUSTER_NAME --namespace syft'
- bash -c './scripts/wait_for.sh service backend --context k3d-$CLUSTER_NAME --namespace syft'
- bash -c './scripts/wait_for.sh service proxy --context k3d-$CLUSTER_NAME --namespace syft'
- bash -c '(kubectl logs service/backend --context k3d-$CLUSTER_NAME --namespace syft -f &) | grep -q "Application startup complete" || true'
+ tox -e dev.k8s.ready -- frontend backend mongo proxy seaweedfs registry
# Run Notebook tests
tox -e e2e.test.notebook
-