feat(baselines) Add Flanders baseline

[edogab33]

Issue

Implements #2595

Description

Implementing FLANDERS baseline into Flower.

[jafermarq]

Hi @edogab33,

Thanks for opening the PR for FLANDERS. I started reviewing it but I noted some parts of the README are not completed yet (which is fine!). Still, I added a couple of comments for the pyproject.toml (one of which will fix the current CI issue)

In addition to that:

• If you'd like to add some plots to your readme, please do so in a new directory named _static placed in the same directory as your README.
• We think it's better not to include the logs from the experiments (I see currently you are including the outputs/ directory.
• Ideally, all the dependencies needed are added to the pyproject.toml so the environment can be built easily by simply doing poetry install. Depending on what packages you use, this might get tricky to achieve actually. If that's the case, it's fine provide additional instructions on how to install those packages after doing poetry install. Then you can install additional ones via the standard pip install <> method.

Ping me once there are some minimal instructions on how to setup the python environment and how to run an experiment. From that I can take a closer look at the code. Happy to discuss about your baseline anytime.

[edogab33]

Thanks for the comments @jafermarq! It's still a WIP, so lot of stuff have to be adjusted :)

If you'd like to add some plots to your readme, please do so in a new directory named _static placed in the same directory as your README.

Got it!

We think it's better not to include the logs from the experiments (I see currently you are including the outputs/ directory.

Yeah, first time using hydra and I didn't realized it created output files. Now everything should be fine (I've disabled the creation of files from base.yaml).

Ideally, all the dependencies needed are added to the pyproject.toml so the environment can be built easily by simply doing poetry install. Depending on what packages you use, this might get tricky to achieve actually. If that's the case, it's fine provide additional instructions on how to install those packages after doing poetry install. Then you can install additional ones via the standard pip install <> method.

I've set up the environment in poetry now. I will check if everything works once I've finished to integrate the code by creating a new environment from scratch.

Ping me once there are some minimal instructions on how to setup the python environment and how to run an experiment. From that I can take a closer look at the code. Happy to discuss about your baseline anytime.

Got it! Thanks again.

[edogab33]

Hi @jafermarq, the baseline should be ready. Let me know if there are any issue or suggestions, I'll take care of them.

[jafermarq]

Hi @edogab33,

Thanks for the updates. I have done a preliminar review of your baseline. See some comments below this message. In addition to those:

• when i ran your code I see quite a few messages of the type: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations. is this expected?
• Don't forget to run the formatting script by: (with your environment activated) cd .., then ./dev/format-baseline.sh flanders. This will automatically correct quite a few small issue, and highlight many more. Generally these are easy to fix.
• Once the above is completed, you'll have to run the ./dev/test-baseline.sh flanders` command. Likely other issues will be highlighted (again, easy to fix for the most part). But do ping me if some of them are not so easy to fix.

I think the main TODO is to change a bit the logic in your main.py so only one experiment runs each time main.py is executed. This helps to keep the code modular (and in line with how other baselines work). Also please use the direcotry hydra creates automatically to save the experiment results (that's where the log will go too). This might require you to change the code you have to generate the plots.

Maybe you want to take a look at how other baselines have done the above. For example: FedProx: https://github.com/adap/flower/tree/main/baselines/fedprox

Let me know if you'd like to discuss how to achieve the above. It's not hard but can't be done in 30mins.

[edogab33]

Hi @jafermarq, everything is ready. However, a test is not passing. Can't get why

[edogab33]

@jafermarq do you know what's the problem with the test?

[jafermarq]

@jafermarq do you know what's the problem with the test?

All seems good now. I'll review soon.

[edogab33]

@jafermarq do you know what's the problem with the test?

All seems good now. I'll review soon.

Thanks! :)

[jafermarq]

Hi @edogab33,

Thank you for for pushing your baseline forward. Sorry for the delay in my review after your updates. Below i leave a couple of comments. Let me know what you think.

I'm now re-running all experiments as you indicate in your Expected Results section. I'll approve the PR shortly after if everything looks alright!

[edogab33]

@jafermarq Thanks for the review! Hope the results are pretty much the same as mine.

I've addressed your issues and realized that if the .sh script is run instead of launching batches of experiments individually, you'll get all results into one file. At the same time, the notebook expects that each experiment has a different one (i.e. all_results_[dataset]_[flanders/no_flanders].csv).
Now the notebook should automatically organize results into the files it expects.

Hope to hear from you soon

[jafermarq]

@jafermarq Thanks for the review! Hope the results are pretty much the same as mine.

I've addressed your issues and realized that if the .sh script is run instead of launching batches of experiments individually, you'll get all results into one file. At the same time, the notebook expects that each experiment has a different one (i.e. all_results_[dataset]_[flanders/no_flanders].csv). Now the notebook should automatically organize results into the files it expects.

Hope to hear from you soon

Thanks! but to generate the results you show in the read me should I run the two commands in the Expected Results section or should I run those in run.sh instead? I'm doing the former but please let me know.

[edogab33]

Thanks! but to generate the results you show in the read me should I run the two commands in the Expected Results section or should I run those in run.sh instead? I'm doing the former but please let me know.

Good point, that's an oversight from my side.

The best way --now that I've updated the notebook to divide the results automatically-- is to run the bash script. For this reason, I removed the former way to run the experiments to avoid mistakes. However, if you've already launched the experiments with that method it's perfectly fine, note that you should run them also for MNIST though.

By the way, as you suggested, I updated the main.py to create the output dir if not present.

@jafermarq

[jafermarq]

It's great to have FLANDERS join Flower baselines! Thanks for the hard work @edogab33!