Merging develop into branch

Signed-off-by: Patrick Foley <[email protected]>
securefederatedai · Apr 26, 2024 · cece780 · cece780
2 parents 5e4b193 + 62fb82b
commit cece780
Show file tree

Hide file tree

Showing 32 changed files with 373 additions and 213 deletions.
diff --git a/README.md b/README.md
@@ -30,20 +30,20 @@ You can simply install OpenFL from PyPI:
 ```
 $ pip install openfl
 ```
-For more installation options check out the [online documentation](https://openfl.readthedocs.io/en/latest/install.html).
+For more installation options check out the [online documentation](https://openfl.readthedocs.io/en/latest/get_started/installation.html).
 
 ## Getting Started
 
 
 OpenFL enables data scientists to set up a federated learning experiment following one of the workflows:
 
-- [Director-based Workflow](https://openfl.readthedocs.io/en/latest/running_the_federation.html#director-based-workflow):
+- [Director-based Workflow](https://openfl.readthedocs.io/en/latest/about/features_index/interactive.html):
 Setup long-lived components to run many experiments in series. Recommended for FL research when many changes to model, dataloader, or hyperparameters are expected
 
-- [Aggregator-based Workflow](https://openfl.readthedocs.io/en/latest/running_the_federation.html#aggregator-based-workflow):
-Define an experiment and distribute it manually. All participants can verify model code and [FL plan](https://openfl.readthedocs.io/en/latest/running_the_federation.html#federated-learning-plan-fl-plan-settings) prior to execution. The federation is terminated when the experiment is finished
+- [Aggregator-based Workflow](https://openfl.readthedocs.io/en/latest/about/features_index/taskrunner.html):
+Define an experiment and distribute it manually. All participants can verify model code and [FL plan](https://openfl.readthedocs.io/en/latest/about/features_index/taskrunner.html#federated-learning-plan-fl-plan-settings) prior to execution. The federation is terminated when the experiment is finished
 
-- [Workflow Interface](https://openfl.readthedocs.io/en/latest/workflow_interface.html) ([*experimental*](https://openfl.readthedocs.io/en/latest/experimental_features.html)):
+- [Workflow Interface](https://openfl.readthedocs.io/en/latest/about/features_index/workflowinterface.html) ([*experimental*](https://openfl.readthedocs.io/en/latest/developer_guide/experimental_features.html)):
 Create complex experiments that extend beyond traditional horizontal federated learning. See the [experimental tutorials](https://github.com/intel/openfl/blob/develop/openfl-tutorials/experimental/) to learn how to coordinate [aggregator validation after collaborator model training](https://github.com/intel/openfl/tree/develop/openfl-tutorials/experimental/Workflow_Interface_102_Aggregator_Validation.ipynb), [perform global differentially private federated learning](https://github.com/psfoley/openfl/tree/experimental-workflow-interface/openfl-tutorials/experimental/Global_DP), measure the amount of private information embedded in a model after collaborator training with [privacy meter](https://github.com/intel/openfl/blob/develop/openfl-tutorials/experimental/Privacy_Meter/readme.md), or [add a watermark to a federated model](https://github.com/intel/openfl/blob/develop/openfl-tutorials/experimental/Workflow_Interface_301_MNIST_Watermarking.ipynb).
 
 The quickest way to test OpenFL is to follow our [tutorials](https://github.com/intel/openfl/tree/develop/openfl-tutorials). </br>
@@ -87,10 +87,10 @@ You can find more details in the following articles:
 ### Supported Aggregation Algorithms
 | Algorithm Name | Paper | PyTorch implementation | TensorFlow implementation | Other frameworks compatibility | How to use | 
 | -------------- | ----- | :--------------------: | :-----------------------: | :----------------------------: | ---------- |
-| FedAvg | [McMahan et al., 2017](https://arxiv.org/pdf/1602.05629.pdf) | ✅ | ✅ | ✅ | [docs](http://openfl.readthedocs.io/en/latest/supported_aggregation_algorithms.html#fedavg) |
-| FedProx | [Li et al., 2020](https://arxiv.org/pdf/1812.06127.pdf) | ✅ | ✅ | ❌ | [docs](http://openfl.readthedocs.io/en/latest/supported_aggregation_algorithms.html#fedprox) |
-| FedOpt | [Reddi et al., 2020](https://arxiv.org/abs/2003.00295) | ✅ | ✅ | ✅ | [docs](http://openfl.readthedocs.io/en/latest/supported_aggregation_algorithms.html#fedopt) |
-| FedCurv | [Shoham et al., 2019](https://arxiv.org/pdf/1910.07796.pdf) | ✅ | ❌ | ❌ | [docs](http://openfl.readthedocs.io/en/latest/supported_aggregation_algorithms.html#fedcurv) |
+| FedAvg | [McMahan et al., 2017](https://arxiv.org/pdf/1602.05629.pdf) | ✅ | ✅ | ✅ | [docs](https://openfl.readthedocs.io/en/latest/about/features.html#aggregation-algorithms) |
+| FedProx | [Li et al., 2020](https://arxiv.org/pdf/1812.06127.pdf) | ✅ | ✅ | ❌ | [docs](https://openfl.readthedocs.io/en/latest/about/features.html#aggregation-algorithms) |
+| FedOpt | [Reddi et al., 2020](https://arxiv.org/abs/2003.00295) | ✅ | ✅ | ✅ | [docs](https://openfl.readthedocs.io/en/latest/about/features.html#aggregation-algorithms) |
+| FedCurv | [Shoham et al., 2019](https://arxiv.org/pdf/1910.07796.pdf) | ✅ | ❌ | ❌ | [docs](https://openfl.readthedocs.io/en/latest/about/features.html#aggregation-algorithms) |
 
 ## Support
 Please join us for our bi-monthly community meetings starting December 1 & 2, 2022! <br>

diff --git a/docs/_static/css/accessibility_overrides.css b/docs/_static/css/accessibility_overrides.css
@@ -196,6 +196,8 @@
   .rst-content .linenodiv pre,
   .rst-content div[class^=highlight] pre,
   .rst-content pre.literal-block{
+    word-break: break-all;
+    white-space: pre-wrap;
     font-size:0.875rem;
   }
 

diff --git a/docs/about/features_index/fed_eval.rst b/docs/about/features_index/fed_eval.rst
@@ -30,7 +30,7 @@ This script can be directly executed as follows:
 
 .. code-block:: console
 
-    python test_hello_federation.py --template torch_cnn_mnist_fed_eval
+    $ python test_hello_federation.py --template torch_cnn_mnist_fed_eval
     
 In order to adapt this template for federated evaluation, the following modifications were made to ``plan.yaml``:
 

diff --git a/docs/about/features_index/interactive.rst b/docs/about/features_index/interactive.rst
@@ -86,7 +86,7 @@ Start the Director on a node with at least two open ports. See :ref:`openfl_ll_c
 
     .. code-block:: console
 
-        fx director create-workspace -p path/to/director_workspace_dir
+        $ fx director create-workspace -p path/to/director_workspace_dir
 
  This workspace will contain received experiments and supplementary files (Director config file and certificates).
 
@@ -100,13 +100,13 @@ Start the Director on a node with at least two open ports. See :ref:`openfl_ll_c
 
     .. code-block:: console
 
-       fx director start --disable-tls -c director_config.yaml
+       $ fx director start --disable-tls -c director_config.yaml
 
  If you have a federation with PKI certificates, run this command.
 
     .. code-block:: console
 
-       fx director start -c director_config.yaml \
+       $ fx director start -c director_config.yaml \
             -rc cert/root_ca.crt \
             -pk cert/priv.key \
             -oc cert/open.crt
@@ -157,7 +157,7 @@ STEP 2: Start the Envoy
 
     .. code-block:: console
 
-        fx envoy create-workspace -p path/to/envoy_workspace_dir
+        $ fx envoy create-workspace -p path/to/envoy_workspace_dir
 
 2. Modify the Envoy config file and local shard descriptor template.
 
@@ -183,7 +183,7 @@ STEP 2: Start the Envoy
 
         ENVOY_NAME=envoy_example_name
 
-        fx envoy start \
+        $ fx envoy start \
             -n "$ENVOY_NAME" \
             --disable-tls \
             --envoy-config-path envoy_config.yaml \
@@ -194,9 +194,9 @@ STEP 2: Start the Envoy
 
     .. code-block:: console
 
-        ENVOY_NAME=envoy_example_name
+        $ ENVOY_NAME=envoy_example_name
 
-        fx envoy start \
+        $ fx envoy start \
             -n "$ENVOY_NAME" \
             --envoy-config-path envoy_config.yaml \
             -dh director_fqdn \

diff --git a/docs/about/features_index/taskrunner.rst b/docs/about/features_index/taskrunner.rst
@@ -149,9 +149,9 @@ STEP 1: Create a Workspace
 
 1.	Start a Python 3.8 (>=3.6, <3.11) virtual environment and confirm |productName| is available.
 
-	.. code-block:: python
+	.. code-block:: console
 
-		fx
+		$ fx
 
 
 2. 	This example uses the :code:`keras_cnn_mnist` template.
@@ -160,8 +160,8 @@ STEP 1: Create a Workspace
 
     .. code-block:: console
 
-        export WORKSPACE_TEMPLATE=keras_cnn_mnist
-        export WORKSPACE_PATH=${HOME}/my_federation
+        $ export WORKSPACE_TEMPLATE=keras_cnn_mnist
+        $ export WORKSPACE_PATH=${HOME}/my_federation
 
 3.	Decide a workspace template, which are end-to-end federated learning training demonstrations. The following is a sample of available templates:
 
@@ -175,14 +175,14 @@ STEP 1: Create a Workspace
 
     .. code-block:: console
 
-       fx workspace create --prefix ${WORKSPACE_PATH}
+       $ fx workspace create --prefix ${WORKSPACE_PATH}
 
 
 4.  Create a workspace directory for the new federation project.
 
     .. code-block:: console
 
-       fx workspace create --prefix ${WORKSPACE_PATH} --template ${WORKSPACE_TEMPLATE}
+       $ fx workspace create --prefix ${WORKSPACE_PATH} --template ${WORKSPACE_TEMPLATE}
 
 
     .. note::
@@ -193,13 +193,13 @@ STEP 1: Create a Workspace
 
     .. code-block:: console
 
-        cd ${WORKSPACE_PATH}
+        $ cd ${WORKSPACE_PATH}
 
 6.  Install the workspace requirements:
 
     .. code-block:: console
 
-        pip install -r requirements.txt
+        $ pip install -r requirements.txt
 
 
 7.	Create an initial set of random model weights.
@@ -213,7 +213,7 @@ STEP 1: Create a Workspace
 
     .. code-block:: console
 
-		fx plan initialize
+		$ fx plan initialize
 
 
     This command initializes the FL plan and auto populates the `fully qualified domain name (FQDN) <https://en.wikipedia.org/wiki/Fully_qualified_domain_name>`_ of the aggregator node. This FQDN is embedded within the FL plan so the collaborator nodes know the address of the externally accessible aggregator server to connect to.
@@ -224,19 +224,19 @@ STEP 1: Create a Workspace
 
 		.. code-block:: console
 
-			fx plan initialize -a aggregator-hostname.internal-domain.com
+			$ fx plan initialize -a aggregator-hostname.internal-domain.com
 
 	- OPTION 2: override the apparent FQDN of the system by setting an FQDN environment variable.
 
 		.. code-block:: console
 
-			export FQDN=x.x.x.x
+			$ export FQDN=x.x.x.x
 
 		and initializing the FL plan
 
 		.. code-block:: console
 
-			fx plan initialize
+			$ fx plan initialize
 
 
 .. note::
@@ -277,49 +277,49 @@ Setting Up the Certificate Authority
 
     .. code-block:: console
 
-       cd WORKSPACE_PATH
+       $ cd WORKSPACE_PATH
 
 2. Set up the aggregator node as the `certificate authority <https://en.wikipedia.org/wiki/Certificate_authority>`_ for the federation.
 
  All certificates will be signed by the aggregator node. Follow the instructions and enter the information as prompted. The command will create a simple database file to keep track of all issued certificates.
 
     .. code-block:: console
 
-       fx workspace certify
+       $ fx workspace certify
 
 3. Run the aggregator certificate creation command, replacing :code:`AFQDN` with the actual `fully qualified domain name (FQDN) <https://en.wikipedia.org/wiki/Fully_qualified_domain_name>`_ for the aggregator node.
 
     .. code-block:: console
 
-       fx aggregator generate-cert-request --fqdn AFQDN
+       $ fx aggregator generate-cert-request --fqdn AFQDN
 
     .. note::
 
        On Linux\*\, you can discover the FQDN with this command:
 
            .. code-block:: console
 
-              hostname --all-fqdns | awk '{print $1}'
+              $ hostname --all-fqdns | awk '{print $1}'
 
    .. note::
 
       You can override the apparent FQDN of the system by setting an FQDN environment variable before creating the certificate.
 
         .. code-block:: console
 
-            fx aggregator generate-cert-request export FQDN=x.x.x.x
+            $ fx aggregator generate-cert-request export FQDN=x.x.x.x
 
       If you omit the :code:`--fdqn` parameter, then :code:`fx` will automatically use the FQDN of the current node assuming the node has been correctly set with a static address.
 
         .. code-block:: console
 
-            fx aggregator generate-cert-request
+            $ fx aggregator generate-cert-request
 
 4. Run the aggregator certificate signing command, replacing :code:`AFQDN` with the actual `fully qualified domain name (FQDN) <https://en.wikipedia.org/wiki/Fully_qualified_domain_name>`_ for the aggregator node.
 
     .. code-block:: console
 
-       fx aggregator certify --fqdn AFQDN
+       $ fx aggregator certify --fqdn AFQDN
 
 
    .. note::
@@ -328,7 +328,7 @@ Setting Up the Certificate Authority
 
         .. code-block:: console
 
-           fx aggregator certify export FQDN=x.x.x.x
+           $ fx aggregator certify export FQDN=x.x.x.x
 
 5. This node now has a signed security certificate as the aggregator for this new federation. You should have the following files.
 
@@ -353,7 +353,7 @@ Exporting the Workspace
 
     .. code-block:: console
 
-       fx workspace export
+       $ fx workspace export
 
    The :code:`export` command will archive the current workspace (with a :code:`zip` file extension) and create a **requirements.txt** of the current Python\*\ packages in the virtual environment.
 
@@ -372,7 +372,7 @@ Importing the Workspace
 
     .. code-block:: console
 
-       fx workspace import --archive WORKSPACE.zip
+       $ fx workspace import --archive WORKSPACE.zip
 
  where **WORKSPACE.zip** is the name of the workspace archive. This will unzip the workspace to the current directory and install the required Python packages within the current virtual environment.
 
@@ -382,8 +382,8 @@ Importing the Workspace
 
     .. code-block:: console
 
-       fx collaborator create -n {COL_LABEL} -d {DATA_PATH:optional}
-       fx collaborator generate-cert-request -n {COL_LABEL}
+       $ fx collaborator create -n {COL_LABEL} -d {DATA_PATH:optional}
+       $ fx collaborator generate-cert-request -n {COL_LABEL}
 
 
  The creation script will also ask you to specify the path to the data. For this example, enter the integer that represents which MNIST shard to use on this collaborator node. For the first collaborator node enter **1**. For the second collaborator node enter **2**.
@@ -405,7 +405,7 @@ Importing the Workspace
 
     .. code-block:: console
 
-       fx collaborator certify --request-pkg /PATH/TO/col_{COL_LABEL}_to_agg_cert_request.zip
+       $ fx collaborator certify --request-pkg /PATH/TO/col_{COL_LABEL}_to_agg_cert_request.zip
 
    where :code:`/PATH/TO/col_{COL_LABEL}_to_agg_cert_request.zip` is the path to the Collaborator CSR Package containing the :code:`.csr` file from the collaborator node. The certificate authority will sign this certificate for use in the federation.
 
@@ -421,7 +421,7 @@ Importing the Workspace
 
     .. code-block:: console
 
-       fx collaborator certify --import /PATH/TO/agg_to_col_{COL_LABEL}_signed_cert.zip
+       $ fx collaborator certify --import /PATH/TO/agg_to_col_{COL_LABEL}_signed_cert.zip
 
 
 
@@ -437,7 +437,7 @@ STEP 3: Start the Federation
 
     .. code-block:: console
 
-       fx aggregator start
+       $ fx aggregator start
 
  Now, the Aggregator is running and waiting for Collaborators to connect.
 
@@ -451,7 +451,7 @@ STEP 3: Start the Federation
 
     .. code-block:: console
 
-       fx collaborator start -n {COLLABORATOR_LABEL}
+       $ fx collaborator start -n {COLLABORATOR_LABEL}
 
     where :code:`COLLABORATOR_LABEL` is the label for this Collaborator.
 
@@ -483,7 +483,7 @@ Among other training artifacts, the aggregator creates the last and best aggrega
 
 .. code-block:: console
 
-    fx model save -i model_protobuf_path.pth -o save_model_path
+    $ fx model save -i model_protobuf_path.pth -o save_model_path
 
 In order for this command to succeed, the **TaskRunner** used in the experiment must implement a :code:`save_native()` method.
 
@@ -522,7 +522,7 @@ Option 1: Deploy a Federation in a Docker Container
 
     .. code-block:: console
 
-       docker run -it --network host openfl
+       $ docker run -it --network host openfl
 
 
 You can now experiment with |productName| in the container. For example, you can test the project pipeline with the `"Hello Federation" bash script <https://github.com/intel/openfl/blob/develop/tests/github/test_hello_federation.sh>`_.
@@ -541,7 +541,7 @@ Option 2: Deploy Your Workspace in a Docker Container
 
     .. code-block:: console
 
-       fx workspace dockerize
+       $ fx workspace dockerize
 
 
     By default, the image is saved as **WORKSPACE_NAME_image.tar** in the workspace directory.

diff --git a/docs/about/license.rst b/docs/about/license.rst
@@ -5,5 +5,5 @@
 License
 ==========
 
-This project is licensed under `Apache License Version 2.0 <https://github.com/kta-intel/openfl/tree/kta/documentation_update>`_. 
+This project is licensed under `Apache License Version 2.0 <https://github.com/securefederatedai/openfl/blob/develop/LICENSE>`_. 
 By contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms.