github.io documentation restructuring (#909)

* updating documentation template Signed-off-by: kta-intel <[email protected]> * WIP quickstart guide Signed-off-by: kta-intel <[email protected]> * documentation updates Signed-off-by: kta-intel <[email protected]> * more updates to getting started Signed-off-by: kta-intel <[email protected]> * more get_started docs Signed-off-by: kta-intel <[email protected]> * getting_started updates Signed-off-by: kta-intel <[email protected]> * documentation overhaul. restructuring github.io Signed-off-by: kta-intel <[email protected]> * lint fix Signed-off-by: kta-intel <[email protected]> * addressing various comments, fixing typos, code snippets, etc Signed-off-by: kta-intel <[email protected]> * add /docs/workflow_interface.rst Signed-off-by: kta-intel <[email protected]> * remove /docs/workflow_interface.rst Signed-off-by: kta-intel <[email protected]> * add add /docs/workflow_interface.rst Signed-off-by: kta-intel <[email protected]> * remove /docs/workflow_interface.rst Signed-off-by: kta-intel <[email protected]> * fixing typo in workflowinterface.rst Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: kta-intel <[email protected]>
securefederatedai · Jul 9, 2024 · a59a827 · a59a827
1 parent 48d5558
commit a59a827
Show file tree

Hide file tree

Showing 56 changed files with 3,231 additions and 1,869 deletions.
diff --git a/docs/_static/css/custom.css b/docs/_static/css/custom.css
@@ -0,0 +1,3 @@
+.toctree-expand { 
+    display: none; 
+}
diff --git a/docs/about/blogs_publications.md b/docs/about/blogs_publications.md
@@ -0,0 +1,11 @@
+Blogs & Publications
+====================
+
+* [Federated learning enables big data for rare cancer boundary detection, Dec 2022](https://www.nature.com/articles/s41467-022-33407-5)
+* [How OpenFL Can Boost Your Federated Learning Project, 2022](https://www.intel.com/content/www/us/en/developer/articles/technical/how-openfl-boost-your-federated-learning-project.html)
+* [OpenFL: the open federated learning library, Oct 2022](https://iopscience.iop.org/article/10.1088/1361-6560/ac97d9/pdf)
+* [Federated Learning With OpenFL for Microservices Applications, Aug 2022](https://blogs.vmware.com/opensource/2022/08/31/federated-learning-with-openfl-for-microservices-applications-2/)
+* [A Path Towards Secure Federated Learning, Apr 2022](https://medium.com/openfl/a-path-towards-secure-federated-learning-c2fb16d5e66e)
+* [Go Federated with OpenFL: Put your Deep Learning pipeline on Federated rails, Oct 2021](https://towardsdatascience.com/go-federated-with-openfl-8bc145a5ead1)
+* [Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data, Jul 2020](https://www.nature.com/articles/s41598-020-69250-1)
+
diff --git a/docs/about/features.rst b/docs/about/features.rst
@@ -0,0 +1,113 @@
+.. # Copyright (C) 2020-2024 Intel Corporation
+.. # SPDX-License-Identifier: Apache-2.0
+
+==========
+Features
+==========
+
+.. _running_a_federation:
+
+---------------------
+Running a Federation
+---------------------
+
+|productName| has multiple options for setting up a federation and running experiments, depending on the users needs. 
+
+Task Runner
+    Define an experiment and distribute it manually. All participants can verify model code and FL plan prior to execution. 
+    The federation is terminated when the experiment is finished. Formerly known as the aggregator-based workflow.
+    `For more info <features_index/taskrunner.html>`_
+
+    .. toctree::
+        :hidden:
+
+        features_index/taskrunner
+
+Interactive
+    Setup long-lived components to run many experiments in series. Recommended for FL research when many changes to model, dataloader, or hyperparameters are expected.
+    Formerly known as the director-based workflow.
+    `For more info <features_index/interactive.html>`_
+
+    .. toctree::
+        :hidden:
+
+        features_index/interactive
+
+Workflow Interface (Experimental)
+    Formulate the experiment as a series of tasks, or a flow. Every flow begins with the start task and concludes with end.
+    Heavily influenced by the interface and design of Netflix's Metaflow, the popular framework for data scientists. 
+    `For more info <features_index/workflowinterface.html>`_
+
+    .. toctree::
+        :hidden:
+
+        features_index/workflowinterface
+
+.. _aggregation_algorithms:
+
+-----------------------
+Aggregation Algorithms
+-----------------------
+
+FedAvg
+    Paper: `McMahan et al., 2017 <https://arxiv.org/pdf/1602.05629.pdf>`_
+    Default aggregation algorithm in |productName|. Multiplies local model weights with relative data size and averages this multiplication result.
+
+FedProx
+    Paper: `Li et al., 2020 <https://arxiv.org/abs/1812.06127>`_
+
+    FedProx in |productName| is implemented as a custom optimizer for PyTorch/TensorFlow. In order to use FedProx, do the following:
+
+    1. PyTorch:
+
+    - replace your optimizer with SGD-based :class:`openfl.utilities.optimizers.torch.FedProxOptimizer` 
+        or Adam-based :class:`openfl.utilities.optimizers.torch.FedProxAdam`.
+        Also, you should save model weights for the next round via calling `.set_old_weights()` method of the optimizer
+        before the training epoch.
+
+    2. TensorFlow:
+
+    - replace your optimizer with SGD-based :py:class:`openfl.utilities.optimizers.keras.FedProxOptimizer`.
+
+    For more details, see :code:`../openfl-tutorials/Federated_FedProx_*_MNIST_Tutorial.ipynb` where * is the framework name.
+
+FedOpt
+    Paper: `Reddi et al., 2020 <https://arxiv.org/abs/2003.00295>`_
+
+    FedOpt in |productName|: :ref:`adaptive_aggregation_functions`
+
+FedCurv
+    Paper: `Shoham et al., 2019 <https://arxiv.org/abs/1910.07796>`_
+
+    Requires PyTorch >= 1.9.0. Other frameworks are not supported yet.
+
+    Use :py:class:`openfl.utilities.fedcurv.torch.FedCurv` to override train function using :code:`.get_penalty()`, :code:`.on_train_begin()`, and :code:`.on_train_end()` methods.
+    In addition, you should override default :code:`AggregationFunction` of the train task with :class:`openfl.interface.aggregation_functions.FedCurvWeightedAverage`.
+    See :code:`PyTorch_Histology_FedCurv` tutorial in :code:`../openfl-tutorials/interactive_api` directory for more details.
+
+.. _federated_evaluation:
+
+---------------------
+Federated Evaluation
+---------------------
+
+Evaluate the accuracy and performance of your model on data distributed across decentralized nodes without comprimising data privacy and security. `For more info <features_index/fed_eval.html>`_
+
+.. toctree::
+    :hidden:
+
+    features_index/fed_eval
+
+.. _privacy_meter:
+
+---------------------
+Privacy Meter
+---------------------
+
+Quantitatively audit data privacy in statistical and machine learning algorithms. `For more info <features_index/privacy_meter.html>`_
+
+.. toctree::
+    :hidden:
+
+    features_index/privacy_meter
+
diff --git a/docs/federated_evaluation.rst → docs/about/features_index/fed_eval.rst b/docs/federated_evaluation.rst → docs/about/features_index/fed_eval.rst
@@ -1,15 +1,18 @@
-Federated Evaluation with OpenFL
-=================================
+.. # Copyright (C) 2020-2024 Intel Corporation
+.. # SPDX-License-Identifier: Apache-2.0
+
+Federated Evaluation with |productName|
+=======================================
 
 Introduction to Federated Evaluation
 -------------------------------------
 
 Model evaluation is an essential part of the machine learning development cycle. In a traditional centralized learning system, all evaluation data is collected on a localized server. Because of this, centralized evaluation of machine learning models is a fairly straightforward task. However, in a federated learning system, data is distributed across multiple decentralized devices or nodes. In an effort to preserve the security and privacy of the distributed data, it is infeasible to simply aggregate all the data into a centralized system. Federated evaluation offers a solution by assessing the model at the client side and aggregating the accuracy without ever having to share the data. This is crucial for ensuring the model's effectiveness and reliability in diverse and real-world environments while respecting privacy and data locality
 
-OpenFL's Support for Federated Evaluation
------------------------------------------
+|productName|'s Support for Federated Evaluation
+-------------------------------------------------
 
-OpenFL, a flexible framework for Federated Learning, has the capability to perform federated evaluation by modifying the federation plan. In this document, we will show how OpenFL can facilitate this process through its task runner API (aggregator-based workflow), where the model evaluation is distributed across various collaborators before being sent to the aggregator. For the task runner API, this involves minor modifications to the ``plan.yaml`` file, which defines the workflow and tasks for the federation. In particular, the federation plan should be defined to run for one forward pass and perform only aggregated model validation
+|productName|, a flexible framework for Federated Learning, has the capability to perform federated evaluation by modifying the federation plan. In this document, we will show how OpenFL can facilitate this process through its task runner API (aggregator-based workflow), where the model evaluation is distributed across various collaborators before being sent to the aggregator. For the task runner API, this involves minor modifications to the ``plan.yaml`` file, which defines the workflow and tasks for the federation. In particular, the federation plan should be defined to run for one forward pass and perform only aggregated model validation
 
 In general pipeline is as follows:
 
@@ -19,7 +22,7 @@ In general pipeline is as follows:
 4. **Aggregation**: The aggregator collects and aggregates these metrics to assess overall model performance.
 
 Example Using the Task Runner API (Aggregator-based Workflow)
--------------------------------------------------------------------
+--------------------------------------------------------------
 
 To demonstrate usage of the task runner API (aggregator-based workflow) for federated evaluation, consider the `Hello Federation example <https://github.com/securefederatedai/openfl/blob/develop/tests/github/test_hello_federation.py>`_. This sample script creates a simple federation with two collaborator nodes and one aggregator node, and executes based on a user specified workspace template. We provide a ``torch_cnn_mnist_fed_eval`` template, which is a federated evaluation template adapted from ``torch_cnn_mnist``. 
 
@@ -28,72 +31,10 @@ This script can be directly executed as follows:
 .. code-block:: console
 
     python test_hello_federation.py --template torch_cnn_mnist_fed_eval
-
+    
 In order to adapt this template for federated evaluation, the following modifications were made to ``plan.yaml``:
 
-.. code-block:: yaml
-
-    # Copyright (C) 2020-2023 Intel Corporation
-    # Licensed subject to the terms of the separately executed evaluation license agreement between Intel Corporation and you.
-    
-    aggregator :
-      defaults : plan/defaults/aggregator.yaml
-      template : openfl.component.Aggregator
-      settings :
-        init_state_path     : save/torch_cnn_mnist_init.pbuf
-        best_state_path     : save/torch_cnn_mnist_best.pbuf
-        last_state_path     : save/torch_cnn_mnist_last.pbuf
-        ########################
-        rounds_to_train     : 1
-        ########################
-        log_metric_callback :
-          template : src.mnist_utils.write_metric
-    
-    collaborator :
-      defaults : plan/defaults/collaborator.yaml
-      template : openfl.component.Collaborator
-      settings :
-        delta_updates    : false
-        opt_treatment    : RESET
-    
-    data_loader :
-      defaults : plan/defaults/data_loader.yaml
-      template : src.ptmnist_inmemory.PyTorchMNISTInMemory
-      settings :
-        collaborator_count : 2
-        data_group_name    : mnist
-        batch_size         : 256
-    
-    task_runner :
-      defaults : plan/defaults/task_runner.yaml
-      template : src.pt_cnn.PyTorchCNN
-    
-    network :
-      defaults : plan/defaults/network.yaml
-    
-    assigner :
-      ########################
-      template : openfl.component.RandomGroupedAssigner
-      settings :
-        task_groups  :
-          - name       : validate
-            percentage : 1.0
-            tasks      :
-              - aggregated_model_validation
-      ########################
-      
-    tasks :
-      ########################
-      aggregated_model_validation:
-        function : validate
-        kwargs   :
-          apply   : global
-          metrics :
-            - acc
-      ########################
-    
-    compression_pipeline :
-      defaults : plan/defaults/compression_pipeline.yaml
+.. literalinclude:: ../../../openfl-workspace/torch_cnn_mnist_fed_eval/plan/plan.yaml
 
 Key Changes for Federated Evaluation:
 
@@ -107,4 +48,4 @@ This sample script will create a federation based on the `torch_cnn_mnist_fed_ev
 
 ---
 
-Congratulations, you have successfully performed federated evaluation across two decentralized collaborator nodes.
+Congratulations, you have successfully performed federated evaluation across two decentralized collaborator nodes.