Merge branch 'main' into driver-retry

.github/ISSUE_TEMPLATE/baseline_request.yml

@@ -41,32 +41,22 @@ body:
         - [ ]  Read the [`first contribution` doc](https://flower.dev/docs/first-time-contributors.html)
         - [ ]  Complete the Flower tutorial
         - [ ]  Read the Flower Baselines docs to get an overview:
-            - [ ]  [https://flower.dev/docs/using-baselines.html](https://flower.dev/docs/using-baselines.html)
-            - [ ]  [https://flower.dev/docs/contributing-baselines.html](https://flower.dev/docs/contributing-baselines.html)
+            - [ ]  [How to use Flower Baselines](https://flower.dev/docs/baselines/how-to-use-baselines.html)
+            - [ ]  [How to contribute a Flower Baseline](https://flower.dev/docs/baselines/how-to-contribute-baselines.html)
   - type: checkboxes
     attributes:
       label: Prepare - understand the scope
       options:
         - label: Read the paper linked above
-        - label: Create the directory structure in Flower Baselines (just the `__init__.py` files and a `README.md`)
-        - label: Before starting to write code, write down all of the specs of this experiment in a README (dataset, partitioning, model, number of clients, all hyperparameters, …)
-        - label: Open a draft PR
+        - label: Decide which experiments you'd like to reproduce. The more the better!
+        - label: Follow the steps outlined in [Add a new Flower Baseline](https://flower.dev/docs/baselines/how-to-contribute-baselines.html#add-a-new-flower-baseline).
+        - label: You can use as reference [other baselines](https://github.com/adap/flower/tree/main/baselines) that the community merged following those steps.
   - type: checkboxes
     attributes:
-      label: Implement - make it work
+      label: Verify your implementation
       options:
-        - label: Implement some form of dataset loading and partitioning in a separate `dataset.py` (doesn’t have to match the paper exactly)
-        - label: Implement the model in PyTorch
-        - label: Write a test that shows that the model has the number of parameters mentioned in the paper
-        - label: Implement the federated learning setup outlined in the paper, maybe starting with fewer clients
-        - label: Plot accuracy and loss
-        - label: Run it and check if the model starts to converge
-  - type: checkboxes
-    attributes:
-      label: Align - make it converge
-      options:
-        - label: Implement the exact data partitioning outlined in the paper
-        - label: Use the exact hyperparameters outlined in the paper
-        - label: Make it converge to roughly the same accuracy that the paper states
-        - label: Commit the final hyperparameters and plots
-        - label: Mark the PR as ready
+        - label: Follow the steps indicated in the `EXTENDED_README.md` that was created in your baseline directory
+        - label: Ensure your code reproduces the results for the experiments you chose
+        - label: Ensure your `README.md` is ready to be run by someone that is no familiar with your code. Are all step-by-step instructions clear?
+        - label: Ensure running the formatting and typing tests for your baseline runs without errors.
+        - label: Clone your repo on a new directory, follow the guide on your own `README.md` and verify everything runs.

.github/workflows/android-release.yml

@@ -16,7 +16,7 @@ jobs:
             working-directory: src/kotlin
     name: Release build and publish
     if: github.repository == 'adap/flower'
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
       - name: Check out code
         uses: actions/checkout@v4

.github/workflows/cpp.yml

@@ -11,7 +11,7 @@ on:
 jobs:
   build_and_test:
     name: Build and test
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
 
     steps:
     - uses: actions/checkout@v4

.github/workflows/e2e.yml

@@ -60,6 +60,8 @@ jobs:
         include:
           - directory: bare
 
+          - directory: bare-https
+
           - directory: jax
 
           - directory: pytorch
@@ -135,7 +137,7 @@ jobs:
       - name: Run virtual client test
         run: python simulation.py
       - name: Run driver test
-        run: ./../test_driver.sh
+        run: ./../test_driver.sh "${{ matrix.directory }}"
 
   strategies:
     runs-on: ubuntu-22.04

.github/workflows/framework-release.yml

@@ -9,7 +9,7 @@ jobs:
   publish:
     if: ${{ github.repository == 'adap/flower' }}
     name: Publish draft
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -53,7 +53,7 @@ jobs:
           cat body.md
           
       - name: Release
-        uses: softprops/action-gh-release@de2c0eb
+        uses: softprops/action-gh-release@v1
         with:
           body_path: ./body.md
           draft: true

.github/workflows/update-pr.yml

@@ -6,7 +6,7 @@ on:
       - 'main'
 jobs:
   autoupdate:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
       - name: Automatically update mergeable PRs
         uses: adRise/[email protected]

README.md

@@ -91,20 +91,26 @@ Stay tuned, more tutorials are coming soon. Topics include **Privacy and Securit
 
 ## Flower Baselines
 
-Flower Baselines is a collection of community-contributed experiments that reproduce the experiments performed in popular federated learning publications. Researchers can build on Flower Baselines to quickly evaluate new ideas:
-
-- [FedAvg](https://arxiv.org/abs/1602.05629):
-  - [MNIST](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedavg_mnist)
-- [FedProx](https://arxiv.org/abs/1812.06127):
-  - [MNIST](https://github.com/adap/flower/tree/main/baselines/fedprox/)
-- [FedBN: Federated Learning on non-IID Features via Local Batch Normalization](https://arxiv.org/abs/2102.07623):
-  - [Convergence Rate](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedbn/convergence_rate)
-- [Adaptive Federated Optimization](https://arxiv.org/abs/2003.00295):
-  - [CIFAR-10/100](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/adaptive_federated_optimization)
-
-Check the Flower documentation to learn more: [Using Baselines](https://flower.dev/docs/baselines/how-to-use-baselines.html)
-
-The Flower community loves contributions! Make your work more visible and enable others to build on it by contributing it as a baseline: [Contributing Baselines](https://flower.dev/docs/baselines/how-to-contribute-baselines.html)
+Flower Baselines is a collection of community-contributed projects that reproduce the experiments performed in popular federated learning publications. Researchers can build on Flower Baselines to quickly evaluate new ideas. The Flower community loves contributions! Make your work more visible and enable others to build on it by contributing it as a baseline!
+
+- [DASHA](https://github.com/adap/flower/tree/main/baselines/dasha)
+- [DepthFL](https://github.com/adap/flower/tree/main/baselines/depthfl)
+- [FedBN](https://github.com/adap/flower/tree/main/baselines/fedbn)
+- [FedMeta](https://github.com/adap/flower/tree/main/baselines/fedmeta)
+- [FedMLB](https://github.com/adap/flower/tree/main/baselines/fedmlb)
+- [FedPer](https://github.com/adap/flower/tree/main/baselines/fedper)
+- [FedProx](https://github.com/adap/flower/tree/main/baselines/fedprox)
+- [FedWav2vec2](https://github.com/adap/flower/tree/main/baselines/fedwav2vec2)
+- [FjORD](https://github.com/adap/flower/tree/main/baselines/fjord)
+- [MOON](https://github.com/adap/flower/tree/main/baselines/moon)
+- [niid-Bench](https://github.com/adap/flower/tree/main/baselines/niid_bench)
+- [TAMUNA](https://github.com/adap/flower/tree/main/baselines/tamuna)
+- [FedAvg](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedavg_mnist)
+- [FedOpt](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/adaptive_federated_optimization)
+
+Please refer to the [Flower Baselines Documentation](https://flower.dev/docs/baselines/) for a detailed categorization of baselines and for additional info including:
+* [How to use Flower Baselines](https://flower.dev/docs/baselines/how-to-use-baselines.html)
+* [How to contribute a new Flower Baseline](https://flower.dev/docs/baselines/how-to-contribute-baselines.html)
 
 ## Flower Usage Examples
 

baselines/README.md

@@ -1,14 +1,14 @@
 # Flower Baselines
 
 
-> We are changing the way we structure the Flower baselines. While we complete the transition to the new format, you can still find the existing baselines in the `flwr_baselines` directory. Currently, you can make use of baselines for [FedAvg](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedavg_mnist), [FedProx](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedprox_mnist), [FedOpt](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/adaptive_federated_optimization), [FedBN](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedbn/convergence_rate), and [LEAF-FEMNIST](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/leaf/femnist). 
+> We are changing the way we structure the Flower baselines. While we complete the transition to the new format, you can still find the existing baselines in the `flwr_baselines` directory. Currently, you can make use of baselines for [FedAvg](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedavg_mnist), [FedOpt](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/adaptive_federated_optimization), and [LEAF-FEMNIST](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/leaf/femnist). 
 
 > The documentation below has been updated to reflect the new way of using Flower baselines.
 
 
 ## Structure
 
-Each baseline in this directory is fully self-contained in terms of source code it its own directory. In addition, each baseline uses its very own Python environment as designed by the contributors of such baseline in order to replicate the experiments in the paper. Each baseline directory contains the following structure:
+Each baseline in this directory is fully self-contained in terms of source code in its own directory. In addition, each baseline uses its very own Python environment as designed by the contributors of such baseline in order to replicate the experiments in the paper. Each baseline directory contains the following structure:
 
 ```bash
 baselines/<baseline-name>/

baselines/baseline_template/README.md

@@ -1,11 +1,11 @@
 ---
 title: title of the paper
 url: URL to the paper page (not the pdf)
-labels: [label1, label2] # please add between 4 and 10 single-word (maybe two-words) labels (e.g. "system heterogeneity", "image classification", "asynchronous", "weight sharing", "cross-silo")
-dataset: [dataset1, dataset2] # list of datasets you include in your baseline
+labels: [label1, label2] # please add between 4 and 10 single-word (maybe two-words) labels (e.g. system heterogeneity, image classification, asynchronous, weight sharing, cross-silo). Do not use ""
+dataset: [dataset1, dataset2] # list of datasets you include in your baseline. Do not use ""
 ---
 
-# :warning:*_Title of your baseline_*
+# :warning: *_Title of your baseline_*
 
 > Note: If you use this baseline in your work, please remember to cite the original authors of the paper as well as the Flower paper.
 
@@ -15,33 +15,33 @@ dataset: [dataset1, dataset2] # list of datasets you include in your baseline
 
 > :warning: Please complete the metadata section at the very top of this README. This generates a table at the top of the file that will facilitate indexing baselines.
 
-****Paper:**** :warning: *_add the URL of the paper page (not to the .pdf). For instance if you link a paper on ArXiv, add here the URL to the abstract page (e.g. https://arxiv.org/abs/1512.03385). If your paper is in from a journal or conference proceedings, please follow the same logic._*
+**Paper:** :warning: *_add the URL of the paper page (not to the .pdf). For instance if you link a paper on ArXiv, add here the URL to the abstract page (e.g. https://arxiv.org/abs/1512.03385). If your paper is in from a journal or conference proceedings, please follow the same logic._*
 
-****Authors:**** :warning: *_list authors of the paper_*
+**Authors:** :warning: *_list authors of the paper_*
 
-****Abstract:**** :warning: *_add here the abstract of the paper you are implementing_*
+**Abstract:** :warning: *_add here the abstract of the paper you are implementing_*
 
 
 ## About this baseline
 
-****What’s implemented:**** :warning: *_Concisely describe what experiment(s) in the publication can be replicated by running the code. Please only use a few sentences. Start with: “The code in this directory …”_*
+**What’s implemented:** :warning: *_Concisely describe what experiment(s) in the publication can be replicated by running the code. Please only use a few sentences. Start with: “The code in this directory …”_*
 
-****Datasets:**** :warning: *_List the datasets you used (if you used a medium to large dataset, >10GB please also include the sizes of the dataset)._*
+**Datasets:** :warning: *_List the datasets you used (if you used a medium to large dataset, >10GB please also include the sizes of the dataset)._*
 
-****Hardware Setup:**** :warning: *_Give some details about the hardware (e.g. a server with 8x V100 32GB and 256GB of RAM) you used to run the experiments for this baseline. Someone out there might not have access to the same resources you have so, could list the absolute minimum hardware needed to run the experiment in a reasonable amount of time ? (e.g. minimum is 1x 16GB GPU otherwise a client model can’t be trained with a sufficiently large batch size). Could you test this works too?_*
+**Hardware Setup:** :warning: *_Give some details about the hardware (e.g. a server with 8x V100 32GB and 256GB of RAM) you used to run the experiments for this baseline. Someone out there might not have access to the same resources you have so, could list the absolute minimum hardware needed to run the experiment in a reasonable amount of time ? (e.g. minimum is 1x 16GB GPU otherwise a client model can’t be trained with a sufficiently large batch size). Could you test this works too?_*
 
-****Contributors:**** :warning: *_let the world know who contributed to this baseline. This could be either your name, your name and affiliation at the time, or your GitHub profile name if you prefer. If multiple contributors signed up for this baseline, please list yourself and your colleagues_*
+**Contributors:** :warning: *_let the world know who contributed to this baseline. This could be either your name, your name and affiliation at the time, or your GitHub profile name if you prefer. If multiple contributors signed up for this baseline, please list yourself and your colleagues_*
 
 
 ## Experimental Setup
 
-****Task:**** :warning: *_what’s the primary task that is being federated? (e.g. image classification, next-word prediction). If you have experiments for several, please list them_*
+**Task:** :warning: *_what’s the primary task that is being federated? (e.g. image classification, next-word prediction). If you have experiments for several, please list them_*
 
-****Model:**** :warning: *_provide details about the model you used in your experiments (if more than use a list). If your model is small, describing it as a table would be :100:. Some FL methods do not use an off-the-shelve model (e.g. ResNet18) instead they create your own. If this is your case, please provide a summary here and give pointers to where in the paper (e.g. Appendix B.4) is detailed._*
+**Model:** :warning: *_provide details about the model you used in your experiments (if more than use a list). If your model is small, describing it as a table would be :100:. Some FL methods do not use an off-the-shelve model (e.g. ResNet18) instead they create your own. If this is your case, please provide a summary here and give pointers to where in the paper (e.g. Appendix B.4) is detailed._*
 
-****Dataset:**** :warning: *_Earlier you listed already the datasets that your baseline uses. Now you should include a breakdown of the details about each of them. Please include information about: how the dataset is partitioned (e.g. LDA with alpha 0.1 as default and all clients have the same number of training examples; or each client gets assigned a different number of samples following a power-law distribution with each client only instances of 2 classes)? if  your dataset is naturally partitioned just state “naturally partitioned”; how many partitions there are (i.e. how many clients)? Please include this an all information relevant about the dataset and its partitioning into a table._*
+**Dataset:** :warning: *_Earlier you listed already the datasets that your baseline uses. Now you should include a breakdown of the details about each of them. Please include information about: how the dataset is partitioned (e.g. LDA with alpha 0.1 as default and all clients have the same number of training examples; or each client gets assigned a different number of samples following a power-law distribution with each client only instances of 2 classes)? if  your dataset is naturally partitioned just state “naturally partitioned”; how many partitions there are (i.e. how many clients)? Please include this an all information relevant about the dataset and its partitioning into a table._*
 
-****Training Hyperparameters:**** :warning: *_Include a table with all the main hyperparameters in your baseline. Please show them with their default value._*
+**Training Hyperparameters:** :warning: *_Include a table with all the main hyperparameters in your baseline. Please show them with their default value._*
 
 
 ## Environment Setup

baselines/baseline_template/baseline_template/main.py

@@ -51,3 +51,7 @@ def main(cfg: DictConfig) -> None:
     # Hydra will generate for you a directory each time you run the code. You
     # can retrieve the path to that directory with this:
     # save_path = HydraConfig.get().runtime.output_dir
+
+
+if __name__ == "__main__":
+    main()

baselines/doc/source/conf.py

@@ -36,7 +36,7 @@
 author = "The Flower Authors"
 
 # The full version, including alpha/beta/rc tags
-release = "1.6.0"
+release = "1.7.0"
 
 
 # -- General configuration ---------------------------------------------------

baselines/doc/source/how-to-use-baselines.rst

@@ -3,7 +3,7 @@ Use Baselines
 
 .. warning::
   We are changing the way we structure the Flower baselines. While we complete the transition to the new format, you can still find the existing baselines and use them: `baselines (old) <https://github.com/adap/flower/tree/main/baselines/flwr_baselines>`_.
-  Currently, you can make use of baselines for `FedAvg <https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedavg_mnist>`_, `FedProx <https://github.com/adap/flower/tree/main/baselines/fedprox>`_, `FedOpt <https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/adaptive_federated_optimization>`_, `FedBN <https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedbn/convergence_rate>`_, and `LEAF-FEMNIST <https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/leaf/femnist>`_.
+  Currently, you can make use of baselines for `FedAvg <https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedavg_mnist>`_, `FedOpt <https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/adaptive_federated_optimization>`_,  and `LEAF-FEMNIST <https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/leaf/femnist>`_.
 
   The documentation below has been updated to reflect the new way of using Flower baselines.
 

baselines/fedbn/.gitignore

@@ -0,0 +1,2 @@
+outputs/
+multirun/