Example for Custom Metrics calculation during Federated Learning

[gubertoli]

Issue

A common question on the flower's questions channel from Slack is related with retrieving other metrics in addition to those available on the ML frameworks.

Description

This PR aims to provide a clear example on how to retrieve custom and additional metrics, to serve as baseline for other people using the Flower for Federated Learning.

Related issues/PRs

N/A

Proposal

A new example on how to calculate custom metrics during Federated Learning.

Explanation

This simple example demonstrate how to calculate custom metrics over multiple clients beyond the traditional ones available in the ML frameworks. In this case, it demonstrate the use of ready-available scikit-learn metrics: accuracy, recall, precision, and f1-score.

Once both the test values (y_test) and the predictions (y_pred) are available on the client side (client.py), other metrics or custom ones are possible to be calculated.

The main takeaways of this implementation are:

• the use of the output_dict on the client side - inside evaluate method on client.py
• the use of the evaluate_metrics_aggregation_fn - to aggregate the metrics on the server side, part of the strategy on server.py

This example is based on the quickstart_tensorflow with CIFAR-10, source here.

Checklist

• Implement proposed change
• Write tests
• Update documentation
• Update changelog
• Make CI checks pass
• Ping maintainers on Slack (channel #contributions)

Any other comments?

N/A

[gubertoli]

@danieljanes
Just a heads up regarding this example, as I missed the history in the Slack channel (+90 days).

[yan-gao-GY]

Hi @gubertoli, thanks for your contribution for this example! I left a few suggested changes and comments.

[gubertoli]

Hi @gubertoli, thanks for your contribution for this example! I left a few suggested changes and comments.

Thank you so much, @yan-gao-GY, for the support!

[danieljanes]

Thanks for the contribution @gubertoli!