refactor(baselines) Update FedRep baseline (#4681)

Co-authored-by: jafermarq <[email protected]>

baselines/fedrep/README.md

@@ -36,91 +36,90 @@ dataset: [CIFAR-10, CIFAR-100]
 
 These two models are modified from the [official repo](https://github.com/rahulv0205/fedrep_experiments)'s. To be clear that, in the official models, there is no BN layers. However, without BN layer helping, training will definitely collapse.
 
-Please see how models are implemented using a so called model_manager and model_split class since FedRep uses head and base layers in a neural network. These classes are defined in the `models.py` file and thereafter called when building new models in the directory `/implemented_models`. Please, extend and add new models as you wish. 
+Please see how models are implemented using a so called model_manager and model_split class since FedRep uses head and base layers in a neural network. These classes are defined in the `models.py` file. Please, extend and add new models as you wish. 
 
 **Dataset:** CIFAR10, CIFAR-100. CIFAR10/100 will be partitioned based on number of classes for data that each client shall receive e.g. 4 allocated classes could be [1, 3, 5, 9].
 
-**Training Hyperparameters:** The hyperparameters can be found in `conf/base.yaml` file which is the configuration file for the main script. 
-
-| Description           | Default Value                       |
-| --------------------- | ----------------------------------- |
-| `num_clients`         | `100`                               |
-| `num_rounds`          | `100`                               |
-| `num_local_epochs`    | `5`                                 |
-| `num_rep_epochs`      | `1`                                 |
-| `enable_finetune`     | `False`                             |
-| `num_finetune_epochs` | `5`                                 |
-| `use_cuda`            | `true`                              |
-| `specified_device`    | `null`                              |
-| `client resources`    | `{'num_cpus': 2, 'num_gpus': 0.5 }` |
-| `learning_rate`       | `0.01`                              |
-| `batch_size`          | `50`                                |
-| `model_name`          | `cnncifar10`                        |
-| `algorithm`           | `fedrep`                            |
+**Training Hyperparameters:** The hyperparameters can be found in `pyproject.toml` file under the `[tool.flwr.app.config]` section.
+
+| Description             | Default Value                       |
+|-------------------------|-------------------------------------|
+| `num-server-rounds`     | `100`                               |
+| `num-local-epochs`      | `5`                                 |
+| `num-rep-epochs`        | `1`                                 |
+| `enable-finetune`       | `False`                             |
+| `num-finetune-epochs`   | `5`                                 |
+| `use-cuda`              | `true`                              |
+| `specified-cuda-device` | `null`                              |
+| `client-resources`      | `{'num-cpus': 2, 'num-gpus': 0.5 }` |
+| `learning-rate`         | `0.01`                              |
+| `batch-size`            | `50`                                |
+| `model-name`            | `cnncifar10`                        |
+| `algorithm`             | `fedrep`                            |
 
 
 ## Environment Setup
 
-To construct the Python environment follow these steps:
+Create a new Python environment using [pyenv](https://github.com/pyenv/pyenv) and [virtualenv plugin](https://github.com/pyenv/pyenv-virtualenv), then install the baseline project:
 
 ```bash
-# Set Python 3.10
-pyenv local 3.10.12
-# Tell poetry to use python 3.10
-poetry env use 3.10.12
+# Create the environment
+pyenv virtualenv 3.10.12 fedrep-env
 
-# Install the base Poetry environment
-poetry install
+# Activate it
+pyenv activate fedrep-env
 
-# Activate the environment
-poetry shell
+# Then install the project
+pip install -e .
 ```
 
 ## Running the Experiments
 
 ```
-python -m fedrep.main # this will run using the default settings in the `conf/base.yaml` 
+flwr run . # this will run using the default settings in the `pyproject.toml` 
 ```
 
-While the config files contain a large number of settings, the ones below are the main ones you'd likely want to modify to .
+While the config files contain a large number of settings, the ones below are the main ones you'd likely want to modify.
 ```bash
-algorithm: fedavg, fedrep # these are currently supported
-dataset.name: cifar10, cifar100
-dataset.num_classes: 2, 5, 20 (only for CIFAR-100)
-model_name: cnncifar10, cnncifar100
+algorithm = "fedavg", "fedrep" # these are currently supported
+dataset-name = "cifar10", "cifar100"
+dataset-split-num-classes = 2, 5, 20 (only for CIFAR-100)
+model-name = "cnncifar10", "cnncifar100"
 ```
-
+See also for instance the configuration files for CIFAR10 and CIFAR100 under the `conf` directory. 
 
 ## Expected Results
+The default algorithm used by all configuration files is `fedrep`. To use `fedavg` please change the `algorithm` property in the respective configuration file. The default federated environment consists of 100 clients.
+
+When the execution completes, a new directory `results` will be created with a json file that contains the running configurations and the results per round. 
+
+> [!NOTE]
+> All plots shown below are generated using the `docs/make_plots.py` script. The script reads all json files generated by the baseline inside the `results` directory.
 
 ### CIFAR-10 (100, 2)
 
 ```
-python -m fedrep.main --config-name cifar10_100_2 algorithm=fedrep 
-python -m fedrep.main --config-name cifar10_100_2 algorithm=fedavg 
+flwr run . --run-config conf/cifar10_2.toml
 ```
 <img src="_static/cifar10_100_2.png" width="400"/> 
 
 ### CIFAR-10 (100, 5)
 
 ```
-python -m fedrep.main --config-name cifar10_100_5 algorithm=fedrep 
-python -m fedrep.main --config-name cifar10_100_5 algorithm=fedavg 
+flwr run . --run-config conf/cifar10_5.toml 
 ```
 <img src="_static/cifar10_100_5.png" width="400"/> 
 
 ### CIFAR-100 (100, 5)
 
 ```
-python -m fedrep.main --config-name cifar100_100_5 algorithm=fedrep 
-python -m fedrep.main --config-name cifar100_100_5 algorithm=fedavg 
+flwr run . --run-config conf/cifar100_5.toml
 ```
 <img src="_static/cifar100_100_5.png" width="400"/> 
 
 ### CIFAR-100 (100, 20)
 
 ```
-python -m fedrep.main --config-name cifar100_100_20 algorithm=fedrep 
-python -m fedrep.main --config-name cifar100_100_20 algorithm=fedavg 
+flwr run . --run-config conf/cifar100_20.toml 
 ```
-<img src="_static/cifar100_100_20.png" width="400"/> 
+<img src="_static/cifar100_100_20.png" width="400"/> 

baselines/fedrep/conf/cifar100_20.toml

@@ -0,0 +1,11 @@
+algorithm = "fedrep"
+
+# model specs
+model-name = "cnncifar100"
+
+# dataset specs
+dataset-name = "cifar100"
+dataset-split = "sample"
+dataset-split-num-classes = 20
+dataset-split-seed = 42
+dataset-split-fraction = 0.83

baselines/fedrep/conf/cifar100_5.toml

@@ -0,0 +1,11 @@
+algorithm = "fedrep"
+
+# model specs
+model-name = "cnncifar100"
+
+# dataset specs
+dataset-name = "cifar100"
+dataset-split = "sample"
+dataset-split-num-classes = 5
+dataset-split-seed = 42
+dataset-split-fraction = 0.83

baselines/fedrep/conf/cifar10_2.toml

@@ -0,0 +1,8 @@
+algorithm = "fedrep"
+
+# dataset specs
+dataset-name = "cifar10"
+dataset-split = "sample"
+dataset-split-num-classes = 2
+dataset-split-seed = 42
+dataset-split-fraction = 0.83

baselines/fedrep/conf/cifar10_5.toml

@@ -0,0 +1,8 @@
+algorithm = "fedrep"
+
+# dataset specs
+dataset-name = "cifar10"
+dataset-split = "sample"
+dataset-split-num-classes = 5
+dataset-split-seed = 42
+dataset-split-fraction = 0.83

baselines/fedrep/docs/make_plots.py

@@ -0,0 +1,50 @@
+"""Generate plots from json files."""
+
+import json
+import os
+from typing import List, Tuple
+
+import matplotlib.pyplot as plt
+
+# Get the current working directory
+DIR = os.path.dirname(os.path.abspath(__file__))
+
+
+def read_from_results(path: str) -> Tuple[str, str, List[float], str, str]:
+    """Load the json file with recorded configurations and results."""
+    with open(path, "r", encoding="UTF-8") as fin:
+        data = json.load(fin)
+        algorithm = data["run_config"]["algorithm"]
+        model = data["run_config"]["model-name"]
+        accuracies = [res["accuracy"] * 100 for res in data["round_res"]]
+        dataset = data["run_config"]["dataset-name"]
+        num_classes = data["run_config"]["dataset-split-num-classes"]
+
+        return algorithm, model, accuracies, dataset, num_classes
+
+
+def make_plot(dir_path: str, plt_title: str) -> None:
+    """Given a directory with json files, generated a plot using the provided title."""
+    plt.figure()
+    with os.scandir(dir_path) as files:
+        for file in files:
+            file_name = os.path.join(dir_path, file.name)
+            print(file_name, flush=True)
+            algo, m, acc, d, n = read_from_results(file_name)
+            rounds = [i + 1 for i in range(len(acc))]
+            print(f"Max accuracy ({algo}): {max(acc):.2f}")
+            plt.plot(rounds, acc, label=f"{algo}-{d}-{n}classes")
+    plt.xlabel("Rounds")
+    plt.ylabel("Accuracy")
+    plt.title(plt_title)
+    plt.grid()
+    plt.legend()
+    plt.savefig(os.path.join(DIR, f"{plt_title}-{algo}"))
+
+
+if __name__ == "__main__":
+    # Plot results generated by the baseline.
+    # Combine them into a full file path.
+    res_dir = os.path.join(DIR, "../results/")
+    title = "Federated Accuracy over Rounds"
+    make_plot(res_dir, plt_title=title)

baselines/fedrep/fedrep/__init__.py

@@ -1 +1 @@
-"""Template baseline package."""
+"""fedrep: A Flower Baseline."""