Federated Runtime + Workspace Builder #876

ParthM-GitHub · 2023-09-28T12:58:06Z

Summary of changes:

interface/experimental.py

Added command to activate experimental OpenFL commands

experimental/component/

Aggregator, & Collaborator component for experimental OpenFL

experimental/federated/

FederatedRuntime experimental workspace configuration manangement

experimental/interface/fl_spec.py

Changes made to support FederatedRuntime

experimental/interface/{keras,torch}/aggregation_functions

WeightedAverage function added for PyTorch, not implemented for Keras

experimental/interface/cli

Added commands for experimental OpenFL

experimental/protocols

GRPC proto files for RPC

experimental/runtime/federated_runtime.py

FederatedRuntime support

experimental/runtime/local_runtime.py

Added function to return collaborator keyword arguments required by WorkspaceBuilder to build data.yaml

experimental/transport/

Added new GRPC functions to support experimental OpenFL

experimental/utilities/resources.py

Removed torch dependency

experimental/workspace_builder

Added workspace_builder module to automatically create FederatedRuntime workspace from LocalRuntime jupyter notebook

openfl-tutorials/experimental/Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb

Added Keras MNIST example with GPU

openfl-tutorials/experimental/requirements_workflow_interface.txt

Added libraries required for WorkspaceBuilder

openfl-workspace/experimental

Tutorial workspaces for FederatedRuntime

openfl-workspace/experimental/template_workspace

Template workspace used by WorkspaceBuilder to generate workspace based on LocalRuntime jupyter notebook

tests/github/experimental/workspace

Added test cases for FederatedRuntime

tests/github/experimental/workspace/test_experimental_agg_based_workflow.py

Script to automatically trigger FederatedRuntime test case

.github/workflows/taskrunner.yml

Added support to trigger testcase_datastore_cli test case automatically by GitHub

Verification performed:

Verified ray memory usage
Verified checkpoints manually
Verified with all testcases, and examples available for LocalRuntime & FederatedRuntime
Verified FederatedRuntime by Running a tutorial on multiple machines

experimental Aggregator and Collaborator in working condition Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Remaning: Integration of data.yaml file. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

2. Fix `fx plan freeze` command not working issue. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

* Initial commit of Running the federation with GaNDLF Documentation Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update README.md Removed references to Intel's ownship, given it's now owned by the LF AI and Data. Signed-off-by: Patrick Foley <[email protected]> * Fix Flake8 C419 for Ubuntu CI (securefederatedai#800) C419 Unnecessary list comprehension passed to any()/all() prevents short-circuiting - rewrite as a generator Signed-off-by: Aleksander Kantak <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update README.md Signed-off-by: Patrick Foley <[email protected]> * Fix warnings and issues in docs (securefederatedai#825) * Fix warnings and issues in docs Signed-off-by: Aleksander Kantak <[email protected]> * fixup! Fix warnings and issues in docs Signed-off-by: Aleksander Kantak <[email protected]> --------- Signed-off-by: Aleksander Kantak <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Add Logo (securefederatedai#827) * Add Logo * Update README.md Signed-off-by: Patrick Foley <[email protected]> * Change OpenFL documentation font to improve accessibility (securefederatedai#809) This replaces the font of OpenFL documents with Intel One Mono font for low vision developers. Known issues: 1. The text font within the images has not been changed. 2. Some icons that do not exist in the new font cannot be displayed properly. Fixes securefederatedai#799 Co-authored-by: Wang, Le <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Jiang, Jiaqiu <[email protected]> Signed-off-by: Li, Qingqing <[email protected]> Signed-off-by: Wang, Le <[email protected]> Signed-off-by: Wu, Caili <[email protected]> Signed-off-by: He, Dan H <[email protected]> Co-authored-by: He, Dan H <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update unit tests to improve code coverage (securefederatedai#821) * Update ci config Signed-off-by: Fang, Xiaoran <[email protected]> * Add unit test for following files - openfl/federated/plan/plan.py - openfl/interface/aggregation_functions/core/adaptive_aggregation.py Signed-off-by: Fang, Xiaoran <[email protected]> * Add some test cases for databases module Signed-off-by: Fang, Xiaoran <[email protected]> * Fix bugs for databases module unittest Signed-off-by: Fang, Xiaoran <[email protected]> * Update unit tests for component module Signed-off-by: Fang, Xiaoran <[email protected]> * Restore workflow config and update some comments Signed-off-by: Fang, Xiaoran <[email protected]> * Enable save_ test case. Add yaml under test dir for unit test usage. Signed-off-by: Fang, Xiaoran <[email protected]> * Remove plan to new dir. Signed-off-by: Fang, Xiaoran <[email protected]> * Remove plan to new dir. Signed-off-by: Fang, Xiaoran <[email protected]> * Add aggregator start test cases. Signed-off-by: Fang, Xiaoran <[email protected]> * Add 2 aggregator test cases. Signed-off-by: Fang, Xiaoran <[email protected]> * Add 1 aggregator test case. Signed-off-by: Fang, Xiaoran <[email protected]> * Format code. Signed-off-by: Fang, Xiaoran <[email protected]> * Refactor code. Signed-off-by: Fang, Xiaoran <[email protected]> * Add collaborator start test cases. Signed-off-by: Fang, Xiaoran <[email protected]> * Add 1 collaborator test case. Signed-off-by: Fang, Xiaoran <[email protected]> * Format with flake8 Signed-off-by: Fang, Xiaoran <[email protected]> * Remove TODO comments Signed-off-by: Fang, Xiaoran <[email protected]> --------- Signed-off-by: Fang, Xiaoran <[email protected]> Co-authored-by: Wang, Wenjie <[email protected]> Co-authored-by: Lei5 Chen <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Add PyTorch linear regression example (securefederatedai#808) This adds a new tutorial example on distributing a linear regression task over OpenFL cluster. The model is defined by Pytorch which is able to run over both cpu (by default) and gpu. The dataset is generated by make_regression from sklearn.datasets with pre-defined parameters. Fixes securefederatedai#797 Co-authored-by: Jiang, Jiaqiu <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Jiang, Jiaqiu <[email protected]> Signed-off-by: Li, Qingqing <[email protected]> Signed-off-by: Wang, Le <[email protected]> Signed-off-by: Wu, Caili <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * This prints out the hash of the CSR to disk for both the aggregator and (securefederatedai#813) * This prints out the hash of the CSR to disk for both the aggregator and collaborator. The user then compares and approves this hash with the hash printed out of the file to validate the CSR. In addition, a warning message is pritned if certify is run in silent mode. Fixes securefederatedai#692 Signed-off-by: Grant Baker <[email protected]> * Refactor read_csr function to use get_csr_hash Signed-off-by: Grant Baker <[email protected]> * Ask to check hashes before prompt --------- Signed-off-by: Grant Baker <[email protected]> Co-authored-by: Grant Baker <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Improve workspace requirements import (securefederatedai#810) Remove the dump_requirement_file operation in export_ method. Fixes securefederatedai#767 Co-authored-by: Li, Qingqing <[email protected]> Co-authored-by: Wu, Caili <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Jiang, Jiaqiu <[email protected]> Signed-off-by: Li, Qingqing <[email protected]> Signed-off-by: Wang, Le <[email protected]> Signed-off-by: Wu, Caili <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Issue 506 Added Example using FedProx (securefederatedai#818) * created new ineractive_api dir to hold pytorch fedprox mnist example corrected files changed to FedProxOptimizer and ran set_old_weights for new FedProx Pytorch example renamed FedProx notebook used mode.parrameters() to get pytorch model weights got weights using state_dict changed old wieghts to list (for serialization) and fixed README input wieghts before zero_grad [Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer. Fixes: securefederatedai#506 Signed-off-by: Klemme, Beverly <[email protected]> Signed-off-by: Baker, Grant <[email protected]> Signed-off-by: ELizabeth Simon, Neethu <[email protected]> Signed-off-by: Jillela, Emmanuel <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> * [Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer. Fixes: securefederatedai#506 Signed-off-by: Klemme, Beverly <[email protected]> Signed-off-by: Baker, Grant <[email protected]> Signed-off-by: Elizabeth Simon, Neethu <[email protected]> Signed-off-by: Jillela, Emmanuel <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> * Update README.md Signed-off-by: Beverly Klemme <[email protected]> * addressed comments by psfoley: corrected words in the jupyter notebook metadata and added a link to the FedProx paper in the README. Signed-off-by: Beverly Klemme <[email protected]> --------- Signed-off-by: Klemme, Beverly <[email protected]> Signed-off-by: Baker, Grant <[email protected]> Signed-off-by: ELizabeth Simon, Neethu <[email protected]> Signed-off-by: Jillela, Emmanuel <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> Signed-off-by: Elizabeth Simon, Neethu <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * [Bug: 768] FX CLI: Separate create, cert gen commands (securefederatedai#807) This change separates existing command "fx collaborator.py generate-cert-request" command into two commands. "fx collaborator create -n {NAME} -d {DATA_PATH: optional}". "fx collaborator generate-cert-request -n {NAME}". Fixes securefederatedai#768 Signed-off-by: Emmanuel Jillela <[email protected]> Co-authored-by: Emmanuel Jillela <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Add new tutorial example to OpenFL interactive API (securefederatedai#812) * Add new tutorial example to OpenFL interactive API This adds a new tutorial example on distributing a linear regression task over OpenFL cluster The model is defined by scikit-learn which is able to run over both cpu (by default) and gpu. The dataset is 1-dimensional noisy data of sinusoid with pre-defined parameters. Fixes securefederatedai#798 Co-authored-by: Beverly Klemme <[email protected]> Co-authored-by: Grant Baker <[email protected]> Signed-off-by: Yi CAO <[email protected]> * reduced requirements.txt in workspace Signed-off-by: Beverly Klemme <[email protected]> --------- Signed-off-by: Yi CAO <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> Co-authored-by: Yi CAO <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * build(deps): bump tensorflow in /openfl-workspace/tf_cnn_histology (securefederatedai#776) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](tensorflow/tensorflow@v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Patrick Foley <[email protected]> * build(deps): bump tensorflow (securefederatedai#777) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](tensorflow/tensorflow@v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Patrick Foley <[email protected]> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <[email protected]> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <[email protected]> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <[email protected]> * Fix documentation links. Change path names and templates for CI Signed-off-by: Patrick Foley <[email protected]> * Fix paths Signed-off-by: Patrick Foley <[email protected]> * Fix paths Signed-off-by: Patrick Foley <[email protected]> * Fix breaking tests Signed-off-by: Patrick Foley <[email protected]> * Add compatible onnx version to requirements.txt file Signed-off-by: Patrick Foley <[email protected]> * Fix wrong csv file name Signed-off-by: Patrick Foley <[email protected]> * Fix wrong csv file name Signed-off-by: Patrick Foley <[email protected]> * Fix wrong names in workflow file Signed-off-by: Patrick Foley <[email protected]> * Fix wrong data path Signed-off-by: Patrick Foley <[email protected]> * Fix lint in test_gandlf.py Signed-off-by: Patrick Foley <[email protected]> * Fix lint errors Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Aleksander Kantak <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Fang, Xiaoran <[email protected]> Signed-off-by: Klemme, Beverly <[email protected]> Signed-off-by: Baker, Grant <[email protected]> Signed-off-by: ELizabeth Simon, Neethu <[email protected]> Signed-off-by: Jillela, Emmanuel <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> Signed-off-by: Elizabeth Simon, Neethu <[email protected]> Signed-off-by: Emmanuel Jillela <[email protected]> Signed-off-by: Yi CAO <[email protected]> Co-authored-by: Sarthak Pati <[email protected]> Co-authored-by: Prashant Shah <[email protected]> Co-authored-by: akantak <[email protected]> Co-authored-by: wangleflex <[email protected]> Co-authored-by: He, Dan H <[email protected]> Co-authored-by: xiaoranf <[email protected]> Co-authored-by: Wang, Wenjie <[email protected]> Co-authored-by: Lei5 Chen <[email protected]> Co-authored-by: Beverly Klemme <[email protected]> Co-authored-by: Emmanuel Jillela <[email protected]> Co-authored-by: Yi CAO <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Parth Mandaliya <[email protected]>

…#841) Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

* Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid Signed-off-by: Patrick Foley <[email protected]> * Fix links in GaNDLF Documentation * Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid Signed-off-by: Patrick Foley <[email protected]> * Fix links in GaNDLF Documentation Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Francis Storr <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

experimental/component/collaborator/collaborator.py file. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

…deratedai#840) Bumps [onnx](https://github.com/onnx/onnx) from 1.12 to 1.13.0. - [Release notes](https://github.com/onnx/onnx/releases) - [Changelog](https://github.com/onnx/onnx/blob/main/docs/Changelog.md) - [Commits](onnx/onnx@v1.12.0...v1.13.0) --- updated-dependencies: - dependency-name: onnx dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Parth Mandaliya <[email protected]>

is work-progress. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

to string "Private attributes: Not Available." Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Updated subset of collaborators testcase Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

… 103, 201, & 301 fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (securefederatedai#833) * fixing FedAvg averaging in order to be compatible with numpy v1.24+ Signed-off-by: kta-intel <[email protected]> * uncommenting installations for consistency with other tutorials Signed-off-by: kta-intel <[email protected]> * fixing 301_MNIST_Watermarking tutorial FedAvg Signed-off-by: kta-intel <[email protected]> * fixing 301_MNIST_Watermarki ng tutorial FedAvg Signed-off-by: kta-intel <[email protected]> * Switching to py38 kernel and clearing cell outputs Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: Parth Mandaliya <[email protected]>

* Initial scans commit for bandit, hadolint, trivy Signed-off-by: Patrick Foley <[email protected]> * Address bandit scan results Signed-off-by: Patrick Foley <[email protected]> * Fix Trivy action Signed-off-by: Patrick Foley <[email protected]> * Fix linting Signed-off-by: Patrick Foley <[email protected]> * Add Coverity Badge Signed-off-by: Patrick Foley <[email protected]> * Update Hadolint threshold to flag errors only Signed-off-by: Patrick Foley <[email protected]> * Update Hadolint threshold to flag errors only Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Parth Mandaliya <[email protected]>

1. Renaming workspace_builder module to workspace_export. 2. WorkspaceBuilder class renamed to WorkspaceExport. 3. Added WorkspaceExport.export function for user which wraps generate_requirements, generate_plan_yaml, generate_data_yaml functions. 4. Merged WorkspaceExport module to `fx workspace create` command by adding new following arguments: - `--notebook` Path to jupyter notebook - `--template_workspace_dir` Output path for generated workspace 5. WorkspaceExport removed from 101, and 103 tutorials. 6. setup.py changed accordingly. Signed-off-by: Parth Mandaliya <[email protected]>

1. output_dir renamed to output_workspace also the user must pass this parameter (default=/tmp is now removed) or there will be an error. 2. template_workspace_path is not required anymore. 3. Added a new tutorials Workflow_Interface_1001_Workspace_Creation_from_JupyterNotebook.ipynb (copy of Workflow_Interface_301_MNIST_Watermarking.ipynb) explaining how to use WorkspaceExport. Signed-off-by: Parth Mandaliya <[email protected]>

…to 5 from 2 Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Parth Mandaliya <[email protected]>

…utrials and 301 mnist watermarking workspace. Sol. imagen library installs param depedency latest version and param package got an upgrde recently. Pinned param to 1.13.0 after installing imagen which fixed the issue. Signed-off-by: Parth Mandaliya <[email protected]>

Signed-off-by: Mansi Sharma <[email protected]>

…on 3.9+. Fix linting, spelling. Remove experimental aggregation functions Signed-off-by: Patrick Foley <[email protected]>

Signed-off-by: Patrick Foley <[email protected]>

* Federated Runtime v1.0 experimental Aggregator and Collaborator in working condition Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added test, and mnist experimental workspaces working prototype Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Working on integration with Ishant's, and Keerti's code Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Second half of integration installment. Remaning: Integration of data.yaml file. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Removing unnecessary directory my_federation. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * 1. Removed unncessart files. 2. Fix `fx plan freeze` command not working issue. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added function for test workspace Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added include_exclude testcase workspace. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Default execution workspace is mnist. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * data.yaml changes are integrated with this update. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Running a federation with GaNDLF Documentation (#794) * Initial commit of Running the federation with GaNDLF Documentation Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update README.md Removed references to Intel's ownship, given it's now owned by the LF AI and Data. Signed-off-by: Patrick Foley <[email protected]> * Fix Flake8 C419 for Ubuntu CI (#800) C419 Unnecessary list comprehension passed to any()/all() prevents short-circuiting - rewrite as a generator Signed-off-by: Aleksander Kantak <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update README.md Signed-off-by: Patrick Foley <[email protected]> * Fix warnings and issues in docs (#825) * Fix warnings and issues in docs Signed-off-by: Aleksander Kantak <[email protected]> * fixup! Fix warnings and issues in docs Signed-off-by: Aleksander Kantak <[email protected]> --------- Signed-off-by: Aleksander Kantak <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Add Logo (#827) * Add Logo * Update README.md Signed-off-by: Patrick Foley <[email protected]> * Change OpenFL documentation font to improve accessibility (#809) This replaces the font of OpenFL documents with Intel One Mono font for low vision developers. Known issues: 1. The text font within the images has not been changed. 2. Some icons that do not exist in the new font cannot be displayed properly. Fixes securefederatedai#799 Co-authored-by: Wang, Le <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Jiang, Jiaqiu <[email protected]> Signed-off-by: Li, Qingqing <[email protected]> Signed-off-by: Wang, Le <[email protected]> Signed-off-by: Wu, Caili <[email protected]> Signed-off-by: He, Dan H <[email protected]> Co-authored-by: He, Dan H <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update unit tests to improve code coverage (#821) * Update ci config Signed-off-by: Fang, Xiaoran <[email protected]> * Add unit test for following files - openfl/federated/plan/plan.py - openfl/interface/aggregation_functions/core/adaptive_aggregation.py Signed-off-by: Fang, Xiaoran <[email protected]> * Add some test cases for databases module Signed-off-by: Fang, Xiaoran <[email protected]> * Fix bugs for databases module unittest Signed-off-by: Fang, Xiaoran <[email protected]> * Update unit tests for component module Signed-off-by: Fang, Xiaoran <[email protected]> * Restore workflow config and update some comments Signed-off-by: Fang, Xiaoran <[email protected]> * Enable save_ test case. Add yaml under test dir for unit test usage. Signed-off-by: Fang, Xiaoran <[email protected]> * Remove plan to new dir. Signed-off-by: Fang, Xiaoran <[email protected]> * Remove plan to new dir. Signed-off-by: Fang, Xiaoran <[email protected]> * Add aggregator start test cases. Signed-off-by: Fang, Xiaoran <[email protected]> * Add 2 aggregator test cases. Signed-off-by: Fang, Xiaoran <[email protected]> * Add 1 aggregator test case. Signed-off-by: Fang, Xiaoran <[email protected]> * Format code. Signed-off-by: Fang, Xiaoran <[email protected]> * Refactor code. Signed-off-by: Fang, Xiaoran <[email protected]> * Add collaborator start test cases. Signed-off-by: Fang, Xiaoran <[email protected]> * Add 1 collaborator test case. Signed-off-by: Fang, Xiaoran <[email protected]> * Format with flake8 Signed-off-by: Fang, Xiaoran <[email protected]> * Remove TODO comments Signed-off-by: Fang, Xiaoran <[email protected]> --------- Signed-off-by: Fang, Xiaoran <[email protected]> Co-authored-by: Wang, Wenjie <[email protected]> Co-authored-by: Lei5 Chen <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Add PyTorch linear regression example (#808) This adds a new tutorial example on distributing a linear regression task over OpenFL cluster. The model is defined by Pytorch which is able to run over both cpu (by default) and gpu. The dataset is generated by make_regression from sklearn.datasets with pre-defined parameters. Fixes #797 Co-authored-by: Jiang, Jiaqiu <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Jiang, Jiaqiu <[email protected]> Signed-off-by: Li, Qingqing <[email protected]> Signed-off-by: Wang, Le <[email protected]> Signed-off-by: Wu, Caili <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * This prints out the hash of the CSR to disk for both the aggregator and (#813) * This prints out the hash of the CSR to disk for both the aggregator and collaborator. The user then compares and approves this hash with the hash printed out of the file to validate the CSR. In addition, a warning message is pritned if certify is run in silent mode. Fixes securefederatedai#692 Signed-off-by: Grant Baker <[email protected]> * Refactor read_csr function to use get_csr_hash Signed-off-by: Grant Baker <[email protected]> * Ask to check hashes before prompt --------- Signed-off-by: Grant Baker <[email protected]> Co-authored-by: Grant Baker <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Improve workspace requirements import (#810) Remove the dump_requirement_file operation in export_ method. Fixes securefederatedai#767 Co-authored-by: Li, Qingqing <[email protected]> Co-authored-by: Wu, Caili <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Jiang, Jiaqiu <[email protected]> Signed-off-by: Li, Qingqing <[email protected]> Signed-off-by: Wang, Le <[email protected]> Signed-off-by: Wu, Caili <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Issue 506 Added Example using FedProx (#818) * created new ineractive_api dir to hold pytorch fedprox mnist example corrected files changed to FedProxOptimizer and ran set_old_weights for new FedProx Pytorch example renamed FedProx notebook used mode.parrameters() to get pytorch model weights got weights using state_dict changed old wieghts to list (for serialization) and fixed README input wieghts before zero_grad [Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer. Fixes: #506 Signed-off-by: Klemme, Beverly <[email protected]> Signed-off-by: Baker, Grant <[email protected]> Signed-off-by: ELizabeth Simon, Neethu <[email protected]> Signed-off-by: Jillela, Emmanuel <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> * [Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer. Fixes: securefederatedai#506 Signed-off-by: Klemme, Beverly <[email protected]> Signed-off-by: Baker, Grant <[email protected]> Signed-off-by: Elizabeth Simon, Neethu <[email protected]> Signed-off-by: Jillela, Emmanuel <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> * Update README.md Signed-off-by: Beverly Klemme <[email protected]> * addressed comments by psfoley: corrected words in the jupyter notebook metadata and added a link to the FedProx paper in the README. Signed-off-by: Beverly Klemme <[email protected]> --------- Signed-off-by: Klemme, Beverly <[email protected]> Signed-off-by: Baker, Grant <[email protected]> Signed-off-by: ELizabeth Simon, Neethu <[email protected]> Signed-off-by: Jillela, Emmanuel <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> Signed-off-by: Elizabeth Simon, Neethu <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * [Bug: 768] FX CLI: Separate create, cert gen commands (#807) This change separates existing command "fx collaborator.py generate-cert-request" command into two commands. "fx collaborator create -n {NAME} -d {DATA_PATH: optional}". "fx collaborator generate-cert-request -n {NAME}". Fixes #768 Signed-off-by: Emmanuel Jillela <[email protected]> Co-authored-by: Emmanuel Jillela <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Add new tutorial example to OpenFL interactive API (#812) * Add new tutorial example to OpenFL interactive API This adds a new tutorial example on distributing a linear regression task over OpenFL cluster The model is defined by scikit-learn which is able to run over both cpu (by default) and gpu. The dataset is 1-dimensional noisy data of sinusoid with pre-defined parameters. Fixes #798 Co-authored-by: Beverly Klemme <[email protected]> Co-authored-by: Grant Baker <[email protected]> Signed-off-by: Yi CAO <[email protected]> * reduced requirements.txt in workspace Signed-off-by: Beverly Klemme <[email protected]> --------- Signed-off-by: Yi CAO <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> Co-authored-by: Yi CAO <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * build(deps): bump tensorflow in /openfl-workspace/tf_cnn_histology (#776) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Patrick Foley <[email protected]> * build(deps): bump tensorflow (#777) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Patrick Foley <[email protected]> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <[email protected]> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <[email protected]> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <[email protected]> * Fix documentation links. Change path names and templates for CI Signed-off-by: Patrick Foley <[email protected]> * Fix paths Signed-off-by: Patrick Foley <[email protected]> * Fix paths Signed-off-by: Patrick Foley <[email protected]> * Fix breaking tests Signed-off-by: Patrick Foley <[email protected]> * Add compatible onnx version to requirements.txt file Signed-off-by: Patrick Foley <[email protected]> * Fix wrong csv file name Signed-off-by: Patrick Foley <[email protected]> * Fix wrong csv file name Signed-off-by: Patrick Foley <[email protected]> * Fix wrong names in workflow file Signed-off-by: Patrick Foley <[email protected]> * Fix wrong data path Signed-off-by: Patrick Foley <[email protected]> * Fix lint in test_gandlf.py Signed-off-by: Patrick Foley <[email protected]> * Fix lint errors Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Aleksander Kantak <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Fang, Xiaoran <[email protected]> Signed-off-by: Klemme, Beverly <[email protected]> Signed-off-by: Baker, Grant <[email protected]> Signed-off-by: ELizabeth Simon, Neethu <[email protected]> Signed-off-by: Jillela, Emmanuel <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> Signed-off-by: Elizabeth Simon, Neethu <[email protected]> Signed-off-by: Emmanuel Jillela <[email protected]> Signed-off-by: Yi CAO <[email protected]> Co-authored-by: Sarthak Pati <[email protected]> Co-authored-by: Prashant Shah <[email protected]> Co-authored-by: akantak <[email protected]> Co-authored-by: wangleflex <[email protected]> Co-authored-by: He, Dan H <[email protected]> Co-authored-by: xiaoranf <[email protected]> Co-authored-by: Wang, Wenjie <[email protected]> Co-authored-by: Lei5 Chen <[email protected]> Co-authored-by: Beverly Klemme <[email protected]> Co-authored-by: Emmanuel Jillela <[email protected]> Co-authored-by: Yi CAO <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Parth Mandaliya <[email protected]> * Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid (#841) Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Fix GaNDLF documentation links (#842) * Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid Signed-off-by: Patrick Foley <[email protected]> * Fix links in GaNDLF Documentation * Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid Signed-off-by: Patrick Foley <[email protected]> * Fix links in GaNDLF Documentation Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Fix incorrectly formatted link in docs (#839) Signed-off-by: Francis Storr <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added licensing information to all experimental workspaces Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Made changes to LocalRuntime, and tested Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Rename collaborator_name variable to name in experimental/component/collaborator/collaborator.py file. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * build(deps): bump onnx in /openfl-workspace/gandlf_seg_test (#840) Bumps [onnx](https://github.com/onnx/onnx) from 1.12 to 1.13.0. - [Release notes](https://github.com/onnx/onnx/releases) - [Changelog](https://github.com/onnx/onnx/blob/main/docs/Changelog.md) - [Commits](https://github.com/onnx/onnx/compare/v1.12.0...v1.13.0) --- updated-dependencies: - dependency-name: onnx dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Parth Mandaliya <[email protected]> * Selected collaborators functionality added, test workspace of the same is work-progress. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Selected collaborator testcases included with this commit. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Testcase private attributes workspace added with this commit. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Instead of remove private attributes from checkpoint, changed the value to string "Private attributes: Not Available." Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added / Modified types of funciton arguments, and their return types. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Aggregator shutdown related changes are done in this commit Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update testflow_subset_of_collaborators.py Updated subset of collaborators testcase Signed-off-by: Parth Mandaliya <[email protected]> * Modified subset of collaborators testcase. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Add files via upload Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Accessibility updates (#861) * Fix incorrectly formatted link in docs Signed-off-by: Francis Storr <[email protected]> * Font styling, color contrast, other accessibility updates This update: 1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples. 2. Adds colors (in `colors.css`) 3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future. Closes #848 Signed-off-by: Francis Storr <[email protected]> --------- Signed-off-by: Francis Storr <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_nlp (#773) * build(deps): bump tensorflow in /openfl-workspace/keras_nlp Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update RMSProp optimizer import Signed-off-by: Patrick Foley <[email protected]> * flake8 Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: Patrick Foley <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_cnn_mnist (#771) * build(deps): bump tensorflow in /openfl-workspace/keras_cnn_mnist Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * revert experimental Adam to legacy (#863) Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: kta-intel <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevin Ta <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_cnn_with_compression (#770) * build(deps): bump tensorflow Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Patrick Foley <[email protected]> * Update Adam Optimizer import Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: Patrick Foley <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * build(deps): bump tensorflow from 2.9.3 to 2.11.1 in /openfl-tutorials/interactive_api/Flax_CNN_CIFAR (#775) * build(deps): bump tensorflow Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Fixed breaking backages Signed-off-by: Patrick Foley <[email protected]> * Add quiet flag back to pip install Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Patrick Foley <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * build(deps): bump tensorflow-cpu from 2.8.4 to 2.11.1 in /openfl-workspace/keras_nlp_gramine_ready (#769) * build(deps): bump tensorflow-cpu Bumps [tensorflow-cpu](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow-cpu dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Use legacy RMSprop optimizer Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Patrick Foley <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Accessibility color contrast fixes (#864) * Fix incorrectly formatted link in docs Signed-off-by: Francis Storr <[email protected]> * Font styling, color contrast, other accessibility updates This update: 1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples. 2. Adds colors (in `colors.css`) 3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future. Closes #848 Signed-off-by: Francis Storr <[email protected]> * Color contrast updates for accessibility Color contrast updates for accessibility - update generic `a` element - update color of links in the toggle-able read-the-docs panel - update the color of the text in search results - update the color of notes headers Signed-off-by: Francis Storr <[email protected]> --------- Signed-off-by: Francis Storr <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Testflow for verifying stdout redirection to Metaflow datastore (#758) * implemented ray.wait * reverted changes back after testing * adding datastore cli test case * removed unused variables * removed stderr validation * fixed lint suggestions Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Tweak link color so it’s not so aggressive (#865) Signed-off-by: Francis Storr <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * External Loop Functionality Added External Loop Functionality Added Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * External Loop Functionality Added External Loop Functionality Added Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * External Loop Functionality Added External Loop Functionality Added Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added following tutorials, and testcase: 1. Vertical FL 2. Vertical FL Two Party 3. Internal Loop Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * plan.yaml modified in 301 watermarking, and subset of collaborators. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Modifying temp.sh for 2 collaborators. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update aggregator.py Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * FQDN switched to localhost FQDN switched to localhost Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * While activating experimental OpenFL commands, install depedencies as well. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Install experimental requirements while activating experimental openfl commands. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update experimental-requirements.txt Added ray 2.2.0 to experimental-requirements.txt file Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update requirements_workflow_interface.txt Metaflow, and Ray dependencies are moved to openfl-workspace/experimental/experimental-requrirements.txt file. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Delete openfl/experimental/utilities/utils.py, and removed torch depency from openfl/experimental/utilities/resources.py Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update requirements_workflow_interface.txt Updated experimental tutorials requirements.txt file Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Updating 201 GPUs tutorial Updated Workflow_Interface_201_Exclusive_GPUs_with_Ray.ipynb for Aggregator, and Collaborator as stateful actor Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Modified 101, 102, 103, 301, 401 tutorials as per Aggregator, & Collaborator stateful collaborators Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Excluding 'experimental' directory from workspace templates Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update 101 tutorial data.yaml Updated 101 tutorials data.yaml to correct a mistake by in index parameter of private attribute function. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update resources.py Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Modified get_number_of_gpus function Modify get_number_of_gpus function Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * This commit brings following changes: 1. Removed torch dependency from "get_number_of_gpus" 2. Added keras mnist example notebook 3. Added keras mnist workspace example 4. Removed openfl-workspace/experimental/experimental-requirements.txt 5. "fx aggregator start" command will install libraries from experimental tutorials requirements_workflow_interface.txt file 6. Added testcase_datastore_cli, testcase_reference and testcase_reference_with_include_exclude 7. Updated 101 torch mnist plan.yaml and data.yaml 8. Removed aggregator start flow thread from cli/aggregator.py 9. At last changes made in federated runtime Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /tests/github/interactive_api_director/experiments/tensorflow_mnist/envoy (#772) * build(deps): bump tensorflow Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Update sd_requirements.txt * revert to legacy SGD and install tensorflow==2.11 for workflow Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: kta-intel <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <[email protected]> Co-authored-by: kta-intel <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/tf_2dunet (#774) * build(deps): bump tensorflow in /openfl-workspace/tf_2dunet Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Update requirements.txt to retrigger CI * Update requirements.txt --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevin Ta <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * This commit brings following changes: 1. Renamed Workflow_Interface_101_Keras_MNIST_with_GPU.ipynb to Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb 2. Updated documentation in Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb 3. protobuf bumped from 3.19.6 to 3.20.0 in setup.py 4. Incorporating review comments (WIP) Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Modified 104 Keras MNIST tutorial Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * This comming brings following changes: 1. Experimental Aggregator workflow testcases moved to test/github/experimental/workspace directory 2. Added dependecy related changes Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Removing unnecessary logs Removing unnecessary logs Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update collaborator.py Removing unnecessary logs Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Avoided deep copying the context by removing private attributes from context before checkpoint Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Incoporated review comments Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added dependecies to each tutorial notebooks Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * pinned tensorboardX (#870) Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * While taking aggregator step remove private attributes before taking checkpoint and set it back to ctx after checkpoint Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Removing unnecessary deepcopies from aggregator.py file Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Removing unnecessary comments Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update stream_redirect.py Removing unnecessary imports Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added color escape sequences to datastore cli localruntime and workspace testcases Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update requirements_privacy_meter.txt torch 1.13.1 requires torchvision 0.14.1 not 0.14.0 Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update Workflow_Interface_VFL_Two_Party Added actor related changed to Workflow_Interface_VFL_Two_Party Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Files added required to test experimental aggregator based workflow by GitHUB actions Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb Removing GPU parameter from collaborator decorator. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update resources.py Updated get_number_of_gpus function Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Updating import statement In 104 keras workspace utils.py updating import SummaryWriter statement from torch to tensorflow Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * With this commit path of template can be explicitely given, user does not have to choose from available example templates. Only applicable to experimental openfl. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Fixed flake8 issues and typos Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Resolved flake8 issues, Added warning statements when data.yaml is not found or aggregator / collaborator private attributes are not found in data.yaml, Tested by running all tutorials and tests Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Moving the logs after data.yaml warning, and removing unnecessary blank lines. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Adding N-level arguments functionality Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update workflow_interface.rst Added section on how to save best and last model from Metaflow CLI Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Resolving last few TODO items. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Updated logs to provide exact information to user. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * This commit brings following updates: 1. Remove unnecessary spaces from Workflow_Interface_102_Aggregator_Validation.ipynb 2. Upgraded protobuf to 3.20.3 in setup.py as per tensorboard requirements 3. Modified 101-torch-cnn-mnist workspace for n-level arguments 4. Added a federated runtime workspace for 102-aggregator-validation Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * This commit brings following changes: 1. Added 501_pytorch_tinyimagenet_transfer_learning workspace for experimental workflow interface 2. Removed `feature_shape` from openfl/experimental/interface/cli/plan.py 3. Resolved flake8 errors in openfl-workspace/experimental directory Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update data.yaml Correcting a mistake train dataset assigned to test and vice versa. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update flow.py Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update flow.py Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Adding functionality to convert jupyter notebook to workspace (in progress) Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Write flow.py, collaborator_private_attrs.py and aggregator_private_attrs.py automatically Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Find aggregator, and collaborator arguments Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Modifying arguments help Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Removed automatically generated Zone.Identifier file Added Approach2 notebook Modified creator.py Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * workspace_creator renamed to workspace_builder finding values without help of any tags Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Finished exporting data.yaml, plan.yaml. Workspace creator validated for 101 MNIST and 301 Watermarking with MNIST Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Removing round number constant argument from 301 MNIST WATERMARKING Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * * Created get_collaborator_kwargs function in localruntime class which takes collaborator name as argument and returns name of private attributes function and arguments required to call it. * Changed workspace builder code used get_collaborator_kwargs function to build data.yaml file. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Modified get_collaborator_args function Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Modified workspace_builder/build.py file to resolve a typo Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Removed unnecessary files from template_workspace directory Moved print_tree to after generating python script Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added documentation and inline comments for workspace_builder Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added libraries required by WorkspaceBuilder Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update Tensorflow, gRPC, Protobuf dependencies (#868) * Update Tensorflow to latest, finally update grpcio/protobuf Signed-off-by: Patrick Foley <[email protected]> * Lint issue fix and missing tf reference Signed-off-by: Patrick Foley <[email protected]> * pyzmq version fixed * fix taskrunner tests for windows Signed-off-by: Mansi Sharma <[email protected]> * fix taskrunner test syntax for windows Signed-off-by: Mansi Sharma <[email protected]> * adding user option to workspace pip install requirements for windows Signed-off-by: Mansi Sharma <[email protected]> * fix windows CI test Signed-off-by: Mansi Sharma <[email protected]> * testing virtual env for windows github actions Signed-off-by: Mansi Sharma <[email protected]> * testing virtual env for windows github actions Signed-off-by: Mansi Sharma <[email protected]> * testing virtual env for windows github actions Signed-off-by: Mansi Sharma <[email protected]> * testing venv for windows Signed-off-by: Mansi Sharma <[email protected]> * test venv for windows * test venv for windows * Added new KerasSerializer. Fixed other Interactive API experiments * Update taskrunner.yml * Update taskrunner.yml * Update workspace.py * Update workspace.py * Update taskrunner.yml * Remove get_model import from global namespace so dependencies are not loaded into memory unnecessarily (breaking windows build) * Refactoring and cleaning up imports to support Windows install * Fixed logger import paths * Fix missing imports * Fix native import * Fix lint errors * Fix keras optimizer patch. Remove irrelevant unit test * Format logs in UTF-8 for windows * Update interactive-kvasir.yml * Consolidate github actions python versions to single file * Update python versions * Update python versions * Update python versions * Reduce # of DataLoader workers for Pytorch Kvasir CI test * Fix Windows encoding * Fix Windows encoding and limit rounds so Github Actions CI doesn't run out of memory Signed-off-by: Patrick Foley <[email protected]> * Fix windows encoding * Fix Windows encoding --------- Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Mansi Sharma <[email protected]> Co-authored-by: Mansi Sharma <[email protected]> Co-authored-by: Mansi Sharma <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Add FL plan description to documentation (#872) * Add plan description to documentation Signed-off-by: Mansi Sharma <[email protected]> * fix indentation Signed-off-by: Mansi Sharma <[email protected]> * Apply suggestions from code review Co-authored-by: Patrick Foley <[email protected]> --------- Signed-off-by: Mansi Sharma <[email protected]> Co-authored-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Resolve Coverity Issues (#874) * Fix coverity issues * Resolve remaining coverity issues Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Migrate to Ubuntu 22.04 LTS release (supported through 2027) (#875) Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Fixed flake8 issues Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Fixed flake8 issues in test/experimental directory Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Accessibility updates (#861) * Fix incorrectly formatted link in docs Signed-off-by: Francis Storr <[email protected]> * Font styling, color contrast, other accessibility updates This update: 1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples. 2. Adds colors (in `colors.css`) 3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future. Closes #848 Signed-off-by: Francis Storr <[email protected]> --------- Signed-off-by: Francis Storr <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (#833) * fixing FedAvg averaging in order to be compatible with numpy v1.24+ Signed-off-by: kta-intel <[email protected]> * uncommenting installations for consistency with other tutorials Signed-off-by: kta-intel <[email protected]> * fixing 301_MNIST_Watermarking tutorial FedAvg Signed-off-by: kta-intel <[email protected]> * fixing 301_MNIST_Watermarki ng tutorial FedAvg Signed-off-by: kta-intel <[email protected]> * Switching to py38 kernel and clearing cell outputs Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: kta-intel <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Testflow for verifying stdout redirection to Metaflow datastore (#758) * implemented ray.wait * reverted changes back after testing * adding datastore cli test case * removed unused variables * removed stderr validation * fixed lint suggestions Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Resolving DCO issue fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (#833) * fixing FedAvg averaging in order to be compatible with numpy v1.24+ Signed-off-by: kta-intel <[email protected]> * uncommenting installations for consistency with other tutorials Signed-off-by: kta-intel <[email protected]> * fixing 301_MNIST_Watermarking tutorial FedAvg Signed-off-by: kta-intel <[email protected]> * fixing 301_MNIST_Watermarki ng tutorial FedAvg Signed-off-by: kta-intel <[email protected]> * Switching to py38 kernel and clearing cell outputs Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: kta-intel <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update EdenPipeline in the documentation (#877) Signed-off-by: Amit Portnoy <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Fixed flake8 issues Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Added weighted_average aggregation function under openfl.experimental.interface.{keras,torch}.aggregation_funtions Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Tested all local-runtime & federated-runtime tutorials and testcases Signed-off-by: Parth Mandaliya <[email protected]> * Resolving merge conflicts in experimental local-runtme tutorials 101, 103, 201, & 301 fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (#833) * fixing FedAvg averaging in order to be compatible with numpy v1.24+ Signed-off-by: kta-intel <[email protected]> * uncommenting installations for consistency with other tutorials Signed-off-by: kta-intel <[email protected]> * fixing 301_MNIST_Watermarking tutorial FedAvg Signed-off-by: kta-intel <[email protected]> * fixing 301_MNIST_Watermarki ng tutorial FedAvg Signed-off-by: kta-intel <[email protected]> * Switching to py38 kernel and clearing cell outputs Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: kta-intel <[email protected]> --------- Signed-off-by: Parth Mandaliya <[email protected]> * WIP: CI Scans (#873) * Initial scans commit for bandit, hadolint, trivy Signed-off-by: Patrick Foley <[email protected]> * Address bandit scan results Signed-off-by: Patrick Foley <[email protected]> * Fix Trivy action Signed-off-by: Patrick Foley <[email protected]> * Fix linting Signed-off-by: Patrick Foley <[email protected]> * Add Coverity Badge Signed-off-by: Patrick Foley <[email protected]> * Update Hadolint threshold to flag errors only Signed-off-by: Patrick Foley <[email protected]> * Update Hadolint threshold to flag errors only Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> * Update ROADMAP.md (#878) Signed-off-by: Parth Mandaliya <[email protected]> * This commit brings following changes: 1. Renaming workspace_builder module to workspace_export. 2. WorkspaceBuilder class renamed to WorkspaceExport. 3. Added WorkspaceExport.export function for user which wraps generate_requirements, generate_plan_yaml, generate_data_yaml functions. 4. Merged WorkspaceExport module to `fx workspace create` command by adding new following arguments: - `--notebook` Path to jupyter notebook - `--template_workspace_dir` Output path for generated workspace 5. WorkspaceExport removed from 101, and 103 tutorials. 6. setup.py changed accordingly. Signed-off-by: Parth Mandaliya <[email protected]> * This commit brings following changes: 1. output_dir renamed to output_workspace also the user must pass this parameter (default=/tmp is now removed) or there will be an error. 2. template_workspace_path is not required anymore. 3. Added a new tutorials Workflow_Interface_1001_Workspace_Creation_from_JupyterNotebook.ipynb (copy of Workflow_Interface_301_MNIST_Watermarking.ipynb) explaining how to use WorkspaceExport. Signed-off-by: Parth Mandaliya <[email protected]> * In Workflow_Interface_1001 tutorial list of collaborator are changed to 5 from 2 Signed-off-by: Parth Mandaliya <[email protected]> * Uncommeted pip install libraries Signed-off-by: Parth Mandaliya <[email protected]> * Adding GPU for aggregator in 201, Global DP & Privacy Meter tutorials Signed-off-by: Parth Mandaliya <[email protected]> * Modifed documentation in 1001 tutorial Signed-off-by: Parth Mandaliya <[email protected]> * Modified documentation in 1001 tutorial Signed-off-by: Parth Mandaliya <[email protected]> * Fixing 'param' attribute not found issue. in 301, 1001 experimental tutrials and 301 mnist watermarking workspace. Sol. imagen library installs param depedency latest version and param package got an upgrde recently. Pinned param to 1.13.0 after installing imagen which fixed the issue. Signed-off-by: Parth Mandaliya <[email protected]> * VFL 2 party example Signed-off-by: Mansi Sharma <[email protected]> * participant names to lowercase and changes to instructions Signed-off-by: Mansi Sharma <[email protected]> * Resolving missing packages from watermarking examples that break python 3.9+. Fix linting, spelling. Remove experimental aggregation functions Signed-off-by: Patrick Foley <[email protected]> * Remove duplicate files in openfl.experimental Signed-off-by: Patrick Foley <[email protected]> * Trigger CI Signed-off-by: Patrick Foley <[email protected]> * Moving test to be conditional Signed-off-by: Patrick Foley <[email protected]> * Fix inconsistent use of enumerate in loops Signed-off-by: Patrick Foley <[email protected]> * Fixing tests Signed-off-by: Patrick Foley <[email protected]> * Accomodate conversion of names to lower case Signed-off-by: Patrick Foley <[email protected]> * Missing torchvision dependency in tests Signed-off-by: Patrick Foley <[email protected]> * Fix lint issues Signed-off-by: Patrick Foley <[email protected]> --------- Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Aleksander Kantak <[email protected]> Signed-off-by: He, Dan H <[email protected]> Signed-off-by: Fang, Xiaoran <[email protected]> Signed-off-by: Klemme, Beverly <[email protected]> Signed-off-by: Baker, Grant <[email protected]> Signed-off-by: ELizabeth Simon, Neethu <[email protected]> Signed-off-by: Jillela, Emmanuel <[email protected]> Signed-off-by: Beverly Klemme <[email protected]> Signed-off-by: Elizabeth Simon, Neethu <[email protected]> Signed-off-by: Emmanuel Jillela <[email protected]> Signed-off-by: Yi CAO <[email protected]> Signed-off-by: Francis Storr <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: kta-intel <[email protected]> Signed-off-by: Mansi Sharma <[email protected]> Signed-off-by: Amit Portnoy <[email protected]> Co-authored-by: Patrick Foley <[email protected]> Co-authored-by: Sarthak Pati <[email protected]> Co-authored-by: Prashant Shah <[email protected]> Co-authored-by: akantak <[email protected]> Co-authored-by: wangleflex <[email protected]> Co-authored-by: He, Dan H <[email protected]> Co-authored-by: xiaoranf <[email protected]> Co-authored-by: Wang, Wenjie <[email protected]> Co-authored-by: Lei5 Chen <[email protected]> Co-authored-by: Beverly Klemme <[email protected]> Co-authored-by: Emmanuel Jillela <[email protected]> Co-authored-by: Yi CAO <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Francis Storr <[email protected]> Co-authored-by: Patrick Foley <[email protected]> Co-authored-by: Kevin Ta <[email protected]> Co-authored-by: Keerti Prakash Talwar <[email protected]> Co-authored-by: kta-intel <[email protected]> Co-authored-by: porteratzo <[email protected]> Co-authored-by: Parth Mandaliya <[email protected]> Co-authored-by: Mansi Sharma <[email protected]> Co-authored-by: Mansi Sharma <[email protected]> Co-authored-by: Amit Portnoy <[email protected]>

ParthMandaliya force-pushed the workspace-creator branch 4 times, most recently from 26eb3e3 to bd18eed Compare October 4, 2023 19:25

ParthM-GitHub and others added 26 commits October 5, 2023 13:34

Federated Runtime v1.0

79fa091

experimental Aggregator and Collaborator in working condition Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Added test, and mnist experimental workspaces working prototype

96016e5

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Working on integration with Ishant's, and Keerti's code

7bf1459

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Second half of integration installment.

c8f1020

Remaning: Integration of data.yaml file. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Removing unnecessary directory my_federation.

039b55b

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

1. Removed unncessart files.

6bb7783

2. Fix `fx plan freeze` command not working issue. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Added function for test workspace

cc811e9

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Added include_exclude testcase workspace.

1c15c64

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Default execution workspace is mnist.

02d986a

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

data.yaml changes are integrated with this update.

0d8ff1d

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid (securefederatedai…

c4b49f4

…#841) Signed-off-by: Patrick Foley <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Fix incorrectly formatted link in docs (securefederatedai#839)

0a2179a

Signed-off-by: Francis Storr <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Added licensing information to all experimental workspaces

1c353b2

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Made changes to LocalRuntime, and tested

2149a0b

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Rename collaborator_name variable to name in

78e6125

experimental/component/collaborator/collaborator.py file. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Selected collaborators functionality added, test workspace of the same

eaf054b

is work-progress. Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Selected collaborator testcases included with this commit.

bf90877

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Testcase private attributes workspace added with this commit.

4e4bc61

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Instead of remove private attributes from checkpoint, changed the value

d3c705d

to string "Private attributes: Not Available." Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Added / Modified types of funciton arguments, and their return types.

bb1c004

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Aggregator shutdown related changes are done in this commit

d415943

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

Update testflow_subset_of_collaborators.py

fd260b9

Updated subset of collaborators testcase Signed-off-by: Parth Mandaliya <[email protected]>

Modified subset of collaborators testcase.

7ca646c

Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]> Signed-off-by: Parth Mandaliya <[email protected]>

kta-intel and others added 3 commits October 5, 2023 13:45

Update ROADMAP.md (securefederatedai#878)

d13c79b

Signed-off-by: Parth Mandaliya <[email protected]>

ParthMandaliya force-pushed the workspace-creator branch from ee072c5 to d13c79b Compare October 5, 2023 08:27

ParthMandaliya and others added 14 commits October 5, 2023 14:01

Merge branch 'develop' into workspace-creator

c8e7257

In Workflow_Interface_1001 tutorial list of collaborator are changed …

adf6f5e

…to 5 from 2 Signed-off-by: Parth Mandaliya <[email protected]>

Uncommeted pip install libraries

d1731de

Signed-off-by: Parth Mandaliya <[email protected]>

Adding GPU for aggregator in 201, Global DP & Privacy Meter tutorials

bfff490

Signed-off-by: Parth Mandaliya <[email protected]>

Modifed documentation in 1001 tutorial

79799aa

Signed-off-by: Parth Mandaliya <[email protected]>

Modified documentation in 1001 tutorial

bff2b4c

Signed-off-by: Parth Mandaliya <[email protected]>

VFL 2 party example

dfaa184

Signed-off-by: Mansi Sharma <[email protected]>

participant names to lowercase and changes to instructions

1b89cbb

Signed-off-by: Mansi Sharma <[email protected]>

Merge branch 'develop' into workspace-creator

1d76c45

Resolving missing packages from watermarking examples that break pyth…

1ad339b

…on 3.9+. Fix linting, spelling. Remove experimental aggregation functions Signed-off-by: Patrick Foley <[email protected]>

Remove duplicate files in openfl.experimental

8a62f2e

Signed-off-by: Patrick Foley <[email protected]>

psfoley added the workflow_interface label Jan 5, 2024

psfoley added 8 commits January 5, 2024 15:11

Trigger CI

fd721b8

Signed-off-by: Patrick Foley <[email protected]>

Moving test to be conditional

f3f3bca

Signed-off-by: Patrick Foley <[email protected]>

Fix inconsistent use of enumerate in loops

023a449

Signed-off-by: Patrick Foley <[email protected]>

Fixing tests

142272d

Signed-off-by: Patrick Foley <[email protected]>

Accomodate conversion of names to lower case

9155f4e

Signed-off-by: Patrick Foley <[email protected]>

Missing torchvision dependency in tests

5bcbbde

Signed-off-by: Patrick Foley <[email protected]>

Bring develop changes into branch

ed6f058

Signed-off-by: Patrick Foley <[email protected]>

Fix lint issues

2aa7adc

Signed-off-by: Patrick Foley <[email protected]>

psfoley approved these changes Mar 1, 2024

View reviewed changes

psfoley merged commit 5071076 into securefederatedai:develop Mar 1, 2024
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Federated Runtime + Workspace Builder #876

Federated Runtime + Workspace Builder #876

ParthM-GitHub commented Sep 28, 2023 •

edited

Loading

Federated Runtime + Workspace Builder #876

Federated Runtime + Workspace Builder #876

Conversation

ParthM-GitHub commented Sep 28, 2023 • edited Loading

ParthM-GitHub commented Sep 28, 2023 •

edited

Loading