Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Federated Runtime + Workspace Builder #876

Merged
merged 145 commits into from
Mar 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
145 commits
Select commit Hold shift + click to select a range
79fa091
Federated Runtime v1.0
ParthM-GitHub Jun 20, 2023
96016e5
Added test, and mnist experimental workspaces working prototype
ParthM-GitHub Jun 27, 2023
7bf1459
Working on integration with Ishant's, and Keerti's code
ParthM-GitHub Jun 28, 2023
c8f1020
Second half of integration installment.
ParthM-GitHub Jun 30, 2023
039b55b
Removing unnecessary directory my_federation.
ParthM-GitHub Jun 30, 2023
6bb7783
1. Removed unncessart files.
ParthM-GitHub Jun 30, 2023
cc811e9
Added function for test workspace
ParthM-GitHub Jul 3, 2023
1c15c64
Added include_exclude testcase workspace.
ParthM-GitHub Jul 3, 2023
02d986a
Default execution workspace is mnist.
ParthM-GitHub Jul 3, 2023
0d8ff1d
data.yaml changes are integrated with this update.
ParthM-GitHub Jul 5, 2023
01a8780
Running a federation with GaNDLF Documentation (#794)
psfoley Jun 29, 2023
c4b49f4
Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid (#841)
psfoley Jun 29, 2023
b999c95
Fix GaNDLF documentation links (#842)
psfoley Jun 29, 2023
0a2179a
Fix incorrectly formatted link in docs (#839)
fstrr Jun 29, 2023
1c353b2
Added licensing information to all experimental workspaces
ParthM-GitHub Jul 5, 2023
2149a0b
Made changes to LocalRuntime, and tested
ParthM-GitHub Jul 6, 2023
78e6125
Rename collaborator_name variable to name in
ParthM-GitHub Jul 10, 2023
1ad809d
build(deps): bump onnx in /openfl-workspace/gandlf_seg_test (#840)
dependabot[bot] Jul 12, 2023
eaf054b
Selected collaborators functionality added, test workspace of the same
ParthM-GitHub Jul 13, 2023
bf90877
Selected collaborator testcases included with this commit.
ParthM-GitHub Jul 14, 2023
4e4bc61
Testcase private attributes workspace added with this commit.
ParthM-GitHub Jul 14, 2023
d3c705d
Instead of remove private attributes from checkpoint, changed the value
ParthM-GitHub Jul 14, 2023
bb1c004
Added / Modified types of funciton arguments, and their return types.
ParthM-GitHub Jul 14, 2023
d415943
Aggregator shutdown related changes are done in this commit
ParthM-GitHub Jul 14, 2023
fd260b9
Update testflow_subset_of_collaborators.py
ParthM-GitHub Jul 17, 2023
7ca646c
Modified subset of collaborators testcase.
ParthM-GitHub Jul 17, 2023
729dc2b
Add files via upload
ParthM-GitHub Jul 17, 2023
6e4bc7b
Accessibility updates (#861)
fstrr Jul 14, 2023
e354f5a
build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspac…
dependabot[bot] Jul 14, 2023
c8218fd
build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspac…
dependabot[bot] Jul 14, 2023
22596b4
build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspac…
dependabot[bot] Jul 14, 2023
853a2c1
build(deps): bump tensorflow from 2.9.3 to 2.11.1 in /openfl-tutorial…
dependabot[bot] Jul 14, 2023
15232f4
build(deps): bump tensorflow-cpu from 2.8.4 to 2.11.1 in /openfl-work…
dependabot[bot] Jul 14, 2023
72ec44f
Accessibility color contrast fixes (#864)
fstrr Jul 14, 2023
b0aad0c
Testflow for verifying stdout redirection to Metaflow datastore (#758)
KeertiX Jul 14, 2023
30a8515
Tweak link color so it’s not so aggressive (#865)
fstrr Jul 14, 2023
b0c055f
External Loop Functionality Added
ParthM-GitHub Jul 19, 2023
4b8b3b1
External Loop Functionality Added
ParthM-GitHub Jul 19, 2023
eb275b2
External Loop Functionality Added
ParthM-GitHub Jul 19, 2023
fe8a0f1
Added following tutorials, and testcase:
ParthM-GitHub Jul 19, 2023
121d5c6
plan.yaml modified in 301 watermarking, and subset of collaborators.
ParthM-GitHub Jul 19, 2023
4ae8b2d
Modifying temp.sh for 2 collaborators.
ParthM-GitHub Jul 19, 2023
49d8567
Update aggregator.py
ParthM-GitHub Jul 19, 2023
4f3db82
FQDN switched to localhost
ParthM-GitHub Jul 19, 2023
ea34ad0
While activating experimental OpenFL commands, install depedencies as…
ParthM-GitHub Jul 20, 2023
26c4e06
Install experimental requirements while activating experimental openf…
ParthM-GitHub Jul 20, 2023
1fff2cc
Update experimental-requirements.txt
ParthM-GitHub Jul 20, 2023
76a0eb8
Update requirements_workflow_interface.txt
ParthM-GitHub Jul 20, 2023
e899e5c
Delete openfl/experimental/utilities/utils.py, and removed torch depe…
ParthM-GitHub Jul 20, 2023
535b617
Update requirements_workflow_interface.txt
ParthM-GitHub Jul 20, 2023
270a549
Updating 201 GPUs tutorial
ParthM-GitHub Jul 20, 2023
78933d9
Modified 101, 102, 103, 301, 401 tutorials as per Aggregator, & Colla…
ParthM-GitHub Jul 20, 2023
5bad4ad
Excluding 'experimental' directory from workspace templates
ParthM-GitHub Jul 20, 2023
6d326e4
Update 101 tutorial data.yaml
ParthM-GitHub Jul 20, 2023
c4e688b
Update resources.py
ParthM-GitHub Jul 21, 2023
d770177
Modified get_number_of_gpus function
ParthM-GitHub Jul 24, 2023
58ec958
This commit brings following changes:
ParthM-GitHub Jul 29, 2023
a16638e
build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /tests/github/in…
dependabot[bot] Jul 25, 2023
ace241d
build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspac…
dependabot[bot] Jul 27, 2023
c182192
This commit brings following changes:
ParthM-GitHub Aug 1, 2023
9c951e5
Modified 104 Keras MNIST tutorial
ParthM-GitHub Aug 2, 2023
002fa20
This comming brings following changes:
ParthM-GitHub Aug 2, 2023
f216691
Removing unnecessary logs
ParthM-GitHub Aug 3, 2023
ef61f9a
Update collaborator.py
ParthM-GitHub Aug 3, 2023
5c52947
Avoided deep copying the context by removing private attributes from …
ParthM-GitHub Aug 3, 2023
3ed3ba9
Incoporated review comments
ParthM-GitHub Aug 3, 2023
0db5f7c
Added dependecies to each tutorial notebooks
ParthM-GitHub Aug 3, 2023
f15eba9
pinned tensorboardX (#870)
porteratzo Aug 4, 2023
eec4de7
While taking aggregator step remove private attributes before taking …
ParthM-GitHub Aug 8, 2023
9960b3c
Removing unnecessary deepcopies from aggregator.py file
ParthM-GitHub Aug 8, 2023
6809e20
Removing unnecessary comments
ParthM-GitHub Aug 8, 2023
2a0117e
Update stream_redirect.py
ParthM-GitHub Aug 9, 2023
76abc9a
Added color escape sequences to datastore cli localruntime and worksp…
ParthM-GitHub Aug 10, 2023
dca8e0b
Update requirements_privacy_meter.txt
ParthM-GitHub Aug 10, 2023
8320f7a
Update Workflow_Interface_VFL_Two_Party
ParthM-GitHub Aug 11, 2023
c9ed4a0
Files added required to test experimental aggregator based workflow b…
ParthM-GitHub Aug 11, 2023
38b9c28
Update Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb
ParthM-GitHub Aug 11, 2023
7b7d2c0
Update resources.py
ParthM-GitHub Aug 11, 2023
e45eab7
Updating import statement
ParthM-GitHub Aug 16, 2023
b460c7a
With this commit path of template can be explicitely given, user does…
ParthM-GitHub Aug 21, 2023
aff94e8
Fixed flake8 issues and typos
ParthM-GitHub Aug 21, 2023
fdb3bc0
Resolved flake8 issues,
ParthM-GitHub Aug 23, 2023
48ac554
Moving the logs after data.yaml warning, and removing unnecessary bla…
ParthM-GitHub Aug 24, 2023
1de9033
Adding N-level arguments functionality
ParthM-GitHub Aug 25, 2023
315b478
Update workflow_interface.rst
ParthM-GitHub Aug 28, 2023
45f40b0
Resolving last few TODO items.
ParthM-GitHub Aug 28, 2023
cd1a07c
Updated logs to provide exact information to user.
ParthM-GitHub Aug 29, 2023
0a831bc
This commit brings following updates:
ParthM-GitHub Aug 30, 2023
e39006c
This commit brings following changes:
ParthM-GitHub Aug 31, 2023
5887172
Update data.yaml
ParthM-GitHub Sep 6, 2023
2be97cd
Update flow.py
ParthM-GitHub Sep 6, 2023
037cff6
Update flow.py
ParthM-GitHub Sep 6, 2023
b6a4b74
Adding functionality to convert jupyter notebook to workspace (in pro…
ParthMandaliya Sep 11, 2023
f554ef9
Write flow.py, collaborator_private_attrs.py and aggregator_private_a…
ParthMandaliya Sep 12, 2023
dcb146d
Find aggregator, and collaborator arguments
ParthMandaliya Sep 12, 2023
4ea327a
Modifying arguments help
ParthMandaliya Sep 12, 2023
483bbc8
Removed automatically generated Zone.Identifier file
ParthMandaliya Sep 13, 2023
5937149
workspace_creator renamed to workspace_builder
ParthM-GitHub Sep 16, 2023
4969561
Finished exporting data.yaml, plan.yaml.
ParthM-GitHub Sep 22, 2023
0f75e50
Removing round number constant argument from 301 MNIST WATERMARKING
ParthM-GitHub Sep 25, 2023
c3e36b6
* Created get_collaborator_kwargs function in localruntime class whic…
ParthM-GitHub Sep 26, 2023
92109a3
Modified get_collaborator_args function
ParthM-GitHub Sep 26, 2023
c16d8a8
Modified workspace_builder/build.py file to resolve a typo
ParthM-GitHub Sep 26, 2023
5160542
Removed unnecessary files from template_workspace directory
ParthM-GitHub Sep 27, 2023
7b5eaec
Added documentation and inline comments for workspace_builder
ParthM-GitHub Sep 28, 2023
23b0b4a
Added libraries required by WorkspaceBuilder
ParthM-GitHub Sep 28, 2023
57dcf45
Update Tensorflow, gRPC, Protobuf dependencies (#868)
psfoley Aug 25, 2023
3fe36fc
Add FL plan description to documentation (#872)
mansishr Aug 29, 2023
1f56583
Resolve Coverity Issues (#874)
psfoley Sep 14, 2023
0358fd3
Migrate to Ubuntu 22.04 LTS release (supported through 2027) (#875)
psfoley Sep 21, 2023
5620734
Fixed flake8 issues
ParthM-GitHub Sep 28, 2023
3ebfe6d
Fixed flake8 issues in test/experimental directory
ParthM-GitHub Sep 28, 2023
3011deb
Accessibility updates (#861)
fstrr Jul 14, 2023
a3bac48
fixing FedAvg in workflow interface tutorials to be compatible with l…
kta-intel Jul 14, 2023
918fd6b
Testflow for verifying stdout redirection to Metaflow datastore (#758)
KeertiX Jul 14, 2023
064d44f
Resolving DCO issue
kta-intel Jul 14, 2023
c0b9e1c
Update EdenPipeline in the documentation (#877)
amitport Sep 28, 2023
c766a91
Fixed flake8 issues
ParthM-GitHub Sep 28, 2023
a331c2a
Added weighted_average aggregation function under openfl.experimental…
ParthM-GitHub Sep 29, 2023
3386c54
Tested all local-runtime & federated-runtime tutorials and testcases
ParthMandaliya Oct 4, 2023
4c11e06
Resolving merge conflicts in experimental local-runtme tutorials 101,…
kta-intel Jul 14, 2023
52c69f4
WIP: CI Scans (#873)
psfoley Sep 29, 2023
d13c79b
Update ROADMAP.md (#878)
psfoley Oct 3, 2023
c8e7257
Merge branch 'develop' into workspace-creator
ParthMandaliya Oct 5, 2023
44e31f6
This commit brings following changes:
ParthMandaliya Oct 6, 2023
c4ded5e
This commit brings following changes:
ParthMandaliya Oct 10, 2023
adf6f5e
In Workflow_Interface_1001 tutorial list of collaborator are changed …
ParthMandaliya Oct 10, 2023
d1731de
Uncommeted pip install libraries
ParthMandaliya Oct 10, 2023
bfff490
Adding GPU for aggregator in 201, Global DP & Privacy Meter tutorials
ParthMandaliya Oct 11, 2023
79799aa
Modifed documentation in 1001 tutorial
ParthMandaliya Oct 11, 2023
bff2b4c
Modified documentation in 1001 tutorial
ParthMandaliya Oct 11, 2023
530d7e3
Fixing 'param' attribute not found issue. in 301, 1001 experimental t…
ParthMandaliya Nov 20, 2023
dfaa184
VFL 2 party example
Dec 19, 2023
1b89cbb
participant names to lowercase and changes to instructions
Dec 20, 2023
1d76c45
Merge branch 'develop' into workspace-creator
psfoley Jan 4, 2024
1ad339b
Resolving missing packages from watermarking examples that break pyth…
psfoley Jan 5, 2024
8a62f2e
Remove duplicate files in openfl.experimental
psfoley Jan 5, 2024
fd721b8
Trigger CI
psfoley Jan 5, 2024
f3f3bca
Moving test to be conditional
psfoley Jan 5, 2024
023a449
Fix inconsistent use of enumerate in loops
psfoley Jan 5, 2024
142272d
Fixing tests
psfoley Jan 5, 2024
9155f4e
Accomodate conversion of names to lower case
psfoley Jan 6, 2024
5bcbbde
Missing torchvision dependency in tests
psfoley Jan 6, 2024
ed6f058
Bring develop changes into branch
psfoley Mar 1, 2024
2aa7adc
Fix lint issues
psfoley Mar 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/experimental_workflow_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,6 @@ jobs:
- name: Workflow - Collaborator Subset (Ray Backend)
run: |
python tests/github/experimental/testflow_subset_of_collaborators.py ray
- name: Test Experimental Aggregator Based Workflow API
run: |
python -m tests.github.experimental.workspace.test_experimental_agg_based_workflow --custom_template tests/github/experimental/workspace/testcase_datastore_cli --fed_workspace aggregator --col col1 --col col2 --rounds-to-train 1
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,4 +126,3 @@ This project is licensed under [Apache License Version 2.0](LICENSE). By contrib
publisher={IOP Publishing}
}
```

2 changes: 1 addition & 1 deletion docs/_static/css/accessibility_overrides.css
Original file line number Diff line number Diff line change
Expand Up @@ -199,4 +199,4 @@
font-size:0.875rem;
}

}
}
5 changes: 3 additions & 2 deletions openfl-tutorials/experimental/Privacy_Meter/cifar10_PM.py
Original file line number Diff line number Diff line change
Expand Up @@ -656,8 +656,9 @@ def end(self):
args = argparser.parse_args()

# Setup participants
# Set `num_gpus=0.0` to `num_gpus=0.3` to run on GPU
aggregator = Aggregator(num_gpus=0.0)
# If running with GPU and 1 GPU is available then
# Set `num_gpus=0.3` to run on GPU
aggregator = Aggregator()

collaborator_names = ["Portland", "Seattle"]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@
"outputs": [],
"source": [
"!pip install git+https://github.com/intel/openfl.git\n",
"!pip install -r ../requirements_workflow_interface.txt"
"!pip install -r ../requirements_workflow_interface.txt\n",
"!pip install torch\n",
"!pip install torchvision"
]
},
{
Expand Down Expand Up @@ -266,7 +268,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.16"
"version": "3.8.17"
}
},
"nbformat": 4,
Expand Down
Loading
Loading