Merge branch 'develop' of https://github.com/theakshaypant/openfl int…

…o fix/gandlf-ci

.github/workflows/tr_docker_gramine_direct.yml

@@ -31,6 +31,11 @@ jobs:
         fx workspace create --prefix example_workspace --template keras_cnn_mnist
         cd example_workspace
         fx plan initialize -a localhost
+        
+        # Disable tensorboard logging as multiprocessing is not supported in Gramine
+        # https://github.com/gramineproject/examples/issues/33
+        sed -i 's/write_logs: true/write_logs: false/g' plan/plan.yaml
+
         fx workspace dockerize --save --revision https://github.com/${GITHUB_REPOSITORY}.git@${{ github.event.pull_request.head.sha }}
 
     - name: Create certificate authority for workspace

CONTRIBUTING.md

@@ -60,7 +60,7 @@ pip install -r linters-requirements.txt
 
 ## Code style
 
-OpenFL uses [black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/) and [precommit](https://pre-commit.com/) to format the code.
+OpenFL uses [ruff](https://github.com/astral-sh/ruff) to lint/format code and [precommit](https://pre-commit.com/) checks.
 
 Run the following command at the **root** directory of the repo to format your code.
 

GOVERNANCE.md

@@ -22,7 +22,7 @@ OpenFL has the following roles in the project:
 
 ### Users
 One of the main OpenFL design principles is a **great usability**. We aim to build an easily to use library for Federated Learning and bring to users a great developer experience. We believe that our users and customers should guide us in building the great software that solves the data silo and data privacy problems. We encourage users to provide us feedback, ask questions, initiate discussions, propose bug reports and suggest features through the following tools and formats:
-- Usage of [OpenFL](https://openfl.readthedocs.io/en/latest/manual.html);
+- Usage of [OpenFL](https://openfl.readthedocs.io/en/latest/developer_guide/manual.html);
 - Discussions in our [Slack channel](https://join.slack.com/t/openfl/shared_invite/zt-ovzbohvn-T5fApk05~YS_iZhjJ5yaTw);
 - Opening [Issues](https://github.com/securefederatedai/openfl/issues) and [discussion topics](https://github.com/securefederatedai/openfl/discussions) in our GitHub;
 - Participation in [Community Meetings](https://github.com/securefederatedai/openfl#support).
@@ -53,14 +53,12 @@ Maintainers’ responsibilities:
 - Participate in discussions through Slack, GitHub or community meetings;
 - Answer the questions and provide reasonable assistance to users and contributors.
 
-See the list of Maintainers [here](https://github.com/securefederatedai/openfl/MAINTAINERS.md). There is no limit on the maximum number of maintainers.
-
 See calendar of the public maintainers meetings [here](https://wiki.lfaidata.foundation/pages/viewpage.action?pageId=70648254).
 
 ### Technical Steering Committee
 The Technical Steering Committee (TSC) is the main decision-making authority of the OpenFL project. The TSC defines the strategic objective of the project, makes business & legal decisions, as well as high-level technical decisions. The TSC consists of elected members. The election takes place every year by the end of the first quarter (Q1) of the year* , but changes in the TSC may be made during the year based on the project needs and the decision by current TSC members.
 
-The current list of TSC members is [here](https://github.com/securefederatedai/openfl/TSC). The number of TSC members can be increased, but up to a reasonable size (no more than 9 people). The minimum size of the TSC should not be less than 3 people.
+The current list of TSC members is [here](https://github.com/securefederatedai/openfl/blob/develop/TSC.md). The number of TSC members can be increased, but up to a reasonable size (no more than 9 people). The minimum size of the TSC should not be less than 3 people.
 
 \* For 2023 there will be no elections, TSC members and Chair are defined by the founding organizations.
 
@@ -84,7 +82,7 @@ The TSC members are responsible for all business, legal and technical oversight
 - Coordinate any *marketing events*, or communications regarding the project.
 
 #### TSC Chair
-The Technical Steering Committee has a Chair. The current TSC Chair is listed [here](https://github.com/securefederatedai/openfl/TSC). The election of a new Chair takes place every year (by the end of Q1 of the year).
+The Technical Steering Committee has a Chair. The current TSC Chair is listed [here](https://github.com/securefederatedai/openfl/blob/develop/TSC.md). The election of a new Chair takes place every year (by the end of Q1 of the year).
 
 TSC Chair’s responsibilities:
 - Serve as the primary *communication contact* between OpenFL and the [LF AI & Data Foundation](https://lfaidata.foundation/);

ROADMAP.md

@@ -44,7 +44,7 @@ The interactive API was originally designed to be a high-level API for OpenFL, b
 
 ### 1.4 Consolidating interfaces
 Today we support three interfaces: TaskRunner, native Python API, and interactive API. These are all distinct APIs, and are not particularly interoperable.
-By the time we reach OpenFL 2.0, our intention is to deprecate the original native [Python API](https://openfl.readthedocs.io/en/latest/source/workflow/running_the_federation.notebook.html) used for simulations, 
+By the time we reach OpenFL 2.0, our intention is to deprecate the original native [Python API](https://openfl.readthedocs.io/en/latest/get_started/examples.html#python-native-api) used for simulations, 
 bring consistency to the remaining interfaces with a high level, middle level, and low level API that are **fully interoperable**. This will result in being able to use the interface you're most comfortable with for a simulation,
 single experiment, or experiment session (with the director / envoy infrastructure).
 
@@ -59,7 +59,7 @@ This causes community fragmentation and distracts from some of the bigger proble
 ## Upcoming OpenFL releases
 
 ### OpenFL 1.6 (Q4 2023)
-1. Use the OpenFL Workflow Interface on distributed infrastructure with the [FederatedRuntime](https://openfl.readthedocs.io/en/latest/workflow_interface.html#runtimes-future-plans)
+1. Use the OpenFL Workflow Interface on distributed infrastructure with the [FederatedRuntime](https://openfl.readthedocs.io/en/latest/about/features_index/workflowinterface.html#runtimes-future-plans)
 2. LLM Support
 3. New use cases enabled by custom workflows
     * Standard ML Models (i.e. Tree-based algorithms)

docs/about/features_index/fed_eval.rst

@@ -32,11 +32,17 @@ This script can be directly executed as follows:
 
     $ python test_hello_federation.py --template torch_cnn_mnist_fed_eval
     
-In order to adapt this template for federated evaluation, the following modifications were made to ``plan.yaml``:
+In order to adapt this template for federated evaluation, the following defaults were added for assigner, aggregator and tasks and same referenced in the ``plan.yaml``:
 
 .. literalinclude:: ../../../openfl-workspace/torch_cnn_mnist_fed_eval/plan/plan.yaml
 
-Key Changes for Federated Evaluation:
+.. literalinclude:: ../../../openfl-workspace/workspace/plan/defaults/federated-evaluation/aggregator.yaml
+
+.. literalinclude:: ../../../openfl-workspace/workspace/plan/defaults/federated-evaluation/assigner.yaml
+
+.. literalinclude:: ../../../openfl-workspace/workspace/plan/defaults/federated-evaluation/tasks_torch.yaml
+
+Key Changes for Federated Evaluation by baking in defaults for:
 
 1. **aggregator.settings.rounds_to_train**: Set to 1
 2. **assigner**: Assign to aggregated_model_validation instead of default assignments
@@ -48,4 +54,4 @@ This sample script will create a federation based on the `torch_cnn_mnist_fed_ev
 
 ---
 
-Congratulations, you have successfully performed federated evaluation across two decentralized collaborator nodes.
+Congratulations, you have successfully performed federated evaluation across two decentralized collaborator nodes with minor default reference changes to plan

docs/structurizer_dsl/README.md

@@ -5,7 +5,7 @@ https://github.com/structurizr/dsl/blob/master/docs/language-reference.md# <br>
 Article: 
 https://dev.to/simonbrown/getting-started-with-the-structurizr-cli-10c2 <br>
 Git:
-https://github.com/structurizr/cli/blob/master/docs/export.md <br>
+https://github.com/structurizr/cli/blob/master/README.md<br>
 
 2. Using Structurizer CLI with Docker: <br>
 https://github.com/aidmax/structurizr-cli-docker <br>

linters-requirements.txt

@@ -1,4 +1,2 @@
-black
-flake8
-isort
 pre-commit
+ruff

openfl-docker/__init__.py

@@ -0,0 +1,2 @@
+# Copyright 2020-2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0

openfl-tutorials/experimental/workflow/1001_Workspace_Creation_from_JupyterNotebook.ipynb

@@ -26,7 +26,7 @@
     "4. User can utilize the experimental `fx` commands to deploy and run the federation seamlessly\n",
     "\n",
     "\n",
-    "The methodology is described using an existing [OpenFL Watermarking Tutorial](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/Workflow_Interface_301_MNIST_Watermarking.ipynb). Let's get started !\n",
+    "The methodology is described using an existing [OpenFL Watermarking Tutorial](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/workflow/301_MNIST_Watermarking.ipynb). Let's get started !\n",
     "\n"
    ]
   },
@@ -747,7 +747,7 @@
     "\n",
     "Users can directly specify a collaborator's private attributes via `collaborator.private_attributes` which is a dictionary where key is name of the attribute and value is the object that is made accessible to collaborator.\n",
     "\n",
-    "For more detailed information on specifying these private attributes, please refer to the first quick start [notebook](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/Workflow_Interface_101_MNIST.ipynb)"
+    "For more detailed information on specifying these private attributes, please refer to the first quick start [notebook](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/workflow/101_MNIST.ipynb)"
    ]
   },
   {

openfl-tutorials/experimental/workflow/101_MNIST.ipynb

@@ -7,7 +7,7 @@
    "metadata": {},
    "source": [
     "# Workflow Interface 101: Quickstart\n",
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/101_MNIST.ipynb)"
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/workflow/101_MNIST.ipynb)"
    ]
   },
   {
@@ -663,7 +663,7 @@
    "metadata": {},
    "source": [
     "# Congratulations!\n",
-    "Now that you've completed your first workflow interface quickstart notebook, see some of the more advanced things you can do in our [other tutorials](broken_link), including:\n",
+    "Now that you've completed your first workflow interface quickstart notebook, see some of the more advanced things you can do in our [other tutorials](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/experimental/workflow), including:\n",
     "\n",
     "- Using the LocalRuntime Ray Backend for dedicated GPU access\n",
     "- Vertical Federated Learning\n",

openfl-tutorials/experimental/workflow/102_Aggregator_Validation.ipynb

@@ -7,7 +7,7 @@
    "metadata": {},
    "source": [
     "# Workflow Interface 102 - Held out aggregator validation\n",
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/Workflow_Interface_102_Aggregator_Validation.ipynb)"
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/workflow/102_Aggregator_Validation.ipynb)"
    ]
   },
   {
@@ -16,7 +16,7 @@
    "id": "bd059520",
    "metadata": {},
    "source": [
-    "In this tutorial, we build on the ideas from the [first](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/101_MNIST.ipynb) quick start notebook, and demonstrate how to perform validation on the aggregator after training."
+    "In this tutorial, we build on the ideas from the [first](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/workflow/101_MNIST.ipynb) quick start notebook, and demonstrate how to perform validation on the aggregator after training."
    ]
   },
   {
@@ -186,7 +186,7 @@
     "scrolled": true
    },
    "source": [
-    "Now we come to the updated flow definition. Here we use the same tasks as the [quickstart](https://github.com/psfoley/openfl/blob/experimental-workflow-interface/openfl-tutorials/experimental/Workflow_Interface_MNIST.ipynb), but give the aggregator a `test_loader` as a private attribute. The aggregator will do a forward pass on each of the aggregator's models using it's validation data, and weight the highest accuracy model higher than others.  "
+    "Now we come to the updated flow definition. Here we use the same tasks as the [quickstart](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/workflow/101_MNIST.ipynb), but give the aggregator a `test_loader` as a private attribute. The aggregator will do a forward pass on each of the aggregator's models using it's validation data, and weight the highest accuracy model higher than others.  "
    ]
   },
   {
@@ -385,7 +385,7 @@
    "metadata": {},
    "source": [
     "# Congratulations!\n",
-    "Now that you've completed your this notebook, see some of the more advanced things you can do in our [other tutorials](broken_link), including:\n",
+    "Now that you've completed your this notebook, see some of the more advanced things you can do in our [other tutorials](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/experimental/workflow), including:\n",
     "\n",
     "- Using the LocalRuntime Ray Backend for dedicated GPU access\n",
     "- Vertical Federated Learning\n",

openfl-tutorials/experimental/workflow/103_Cyclic_Institutional_Incremental_Learning.ipynb

@@ -7,7 +7,7 @@
    "metadata": {},
    "source": [
     "# Workflow Interface 103 - Cyclic Institutional Incremental Learning\n",
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/Workflow_Interface_103_Cyclic_Institutional_Incremental_Learning.ipynb)"
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/workflow/103_Cyclic_Institutional_Incremental_Learning.ipynb)"
    ]
   },
   {
@@ -195,7 +195,7 @@
     "scrolled": true
    },
    "source": [
-    "Now we come to the updated flow definition. Here we use the same tasks as the [quickstart](https://github.com/psfoley/openfl/blob/experimental-workflow-interface/openfl-tutorials/experimental/Workflow_Interface_MNIST.ipynb), but give the aggregator a `test_loader` as a private attribute. The aggregator will do a forward pass on each of the aggregator's models using it's validation data, and weight the highest accuracy model higher than others.  "
+    "Now we come to the updated flow definition. Here we use the same tasks as the [quickstart](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/workflow/101_MNIST.ipynb), but give the aggregator a `test_loader` as a private attribute. The aggregator will do a forward pass on each of the aggregator's models using it's validation data, and weight the highest accuracy model higher than others.  "
    ]
   },
   {
@@ -658,7 +658,7 @@
    "metadata": {},
    "source": [
     "# Congratulations!\n",
-    "Now that you've completed your this notebook, see some of the more advanced things you can do in our [other tutorials](broken_link), including:\n",
+    "Now that you've completed your this notebook, see some of the more advanced things you can do in our [other tutorials](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/experimental/workflow), including:\n",
     "\n",
     "- Using the LocalRuntime Ray Backend for dedicated GPU access\n",
     "- Vertical Federated Learning\n",

openfl-tutorials/experimental/workflow/104_Keras_MNIST_with_CPU.ipynb

@@ -6,7 +6,7 @@
    "metadata": {},
    "source": [
     "# # Workflow Interface 104: Working with Keras on CPU\n",
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/snehal-das/openfl/blob/develop/openfl-tutorials/experimental/104_Keras_MNIST_with_CPU.ipynb)\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/workflow/104_Keras_MNIST_with_CPU.ipynb)\n",
     "\n",
     "## Training a CNN on CPU using the Workflow Interface and MNIST data.\n",
     "\n",

openfl-tutorials/experimental/workflow/104_Keras_MNIST_with_GPU.ipynb

@@ -5,7 +5,7 @@
    "metadata": {},
    "source": [
     "# Workflow Interface 104: Working with Keras\n",
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/104_Keras_MNIST_with_GPU.ipynb)"
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/workflow/104_Keras_MNIST_with_GPU.ipynb)"
    ]
   },
   {
@@ -323,7 +323,7 @@
    "metadata": {},
    "source": [
     "# Congratulations!\n",
-    "Now that you've completed your this notebook, see some of the more advanced things you can do in our [other tutorials](broken_link), including:\n",
+    "Now that you've completed your this notebook, see some of the more advanced things you can do in our [other tutorials](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/experimental/workflow), including:\n",
     "\n",
     "- Using the LocalRuntime Ray Backend for dedicated GPU access\n",
     "- Vertical Federated Learning\n",

openfl-tutorials/experimental/workflow/104_MNIST_XPU.ipynb

@@ -6,7 +6,7 @@
    "metadata": {},
    "source": [
     "# Workflow Interface 104: MNIST XPU\n",
-    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/104_MNIST_XPU.ipynb)"
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/experimental/workflow/104_MNIST_XPU.ipynb)"
    ]
   },
   {
@@ -701,7 +701,7 @@
    "metadata": {},
    "source": [
     "# Congratulations!\n",
-    "Now that you've completed your first workflow interface quickstart notebook, see some of the more advanced things you can do in our [other tutorials](broken_link), including:\n",
+    "Now that you've completed your first workflow interface quickstart notebook, see some of the more advanced things you can do in our [other tutorials](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/experimental/workflow), including:\n",
     "\n",
     "- Using the LocalRuntime Ray Backend for dedicated GPU access\n",
     "- Vertical Federated Learning\n",