Merge branch 'securefederatedai:develop' into straggler_handling_update

.pre-commit-config.yaml

@@ -6,7 +6,6 @@ repos:
     hooks:
       - id: end-of-file-fixer
         files: \.py$
-      - id: requirements-txt-fixer
       - id: check-merge-conflict
       - id: check-case-conflict
       - id: check-json

.readthedocs.yaml

@@ -1,37 +1,23 @@
-# Read the Docs configuration file for Sphinx projects
+# .readthedocs.yml
+# Read the Docs configuration file
 # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
-
-# Required
 version: 2
 
-# Set the OS, Python version and other tools you might need
 build:
-  os: ubuntu-22.04
+  os: "ubuntu-22.04"
   tools:
     python: "3.10"
-    # You can also specify other tool versions:
-    # nodejs: "20"
-    # rust: "1.70"
-    # golang: "1.20"
 
-# Build documentation in the "docs/" directory with Sphinx
+# Build documentation in the docs/ directory with Sphinx
 sphinx:
   configuration: docs/conf.py
-  # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
-  # builder: "dirhtml"
-  # Fail on all warnings to avoid broken references
-  # fail_on_warning: true
+  fail_on_warning: false
 
 # Optionally build your docs in additional formats such as PDF and ePub
-# formats:
-#   - pdf
-#   - epub
+formats:
+  - htmlzip
 
-# Optional but recommended, declare the Python requirements required
-# to build your documentation
-# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+# Optionally set the version of Python and requirements required to build your docs
 python:
   install:
-    - method: pip
-      path: .
-    - requirements: docs/docs-requirements.txt
+    - requirements: docs/requirements.txt

CONTRIBUTING.md

@@ -1,79 +1,2 @@
 # Contributing to OpenFL
-
-We welcome contributions from the community. There are several ways to contribute:
-* Improvements in [documentation](https://openfl.readthedocs.io/en/latest/).
-* Contributing to OpenFL's code-base: via bug-fixes or feature additions.
-* Answering questions on our [discussions page](https://github.com/securefederatedai/openfl/discussions).
-* Participating in our [roadmap](https://github.com/securefederatedai/openfl/blob/develop/ROADMAP.md) discussions.
-
-We have a slack [channel](https://join.slack.com/t/openfl/shared_invite/zt-ovzbohvn-T5fApk05~YS_iZhjJ5yaTw) and we host regular [community meetings](https://github.com/securefederatedai/openfl#support).
-
-
-## How to contribute code
-### Step 1. Open an issue
-
-Before you start making any changes, it is always good to open an [issue](https://github.com/securefederatedai/openfl/issues/new/choose) first (assuming one does not already exist), outlining your proposed changes. We can give you feedback, and potentially validate the proposed changes.
-
-For minor changes (akin to a documentation or bug fix), proceed to opening a Pull Request (PR) directly.
-
-### Step 2. Make code changes
-
-To modify code, you need to fork the repository. Set up a development environment as covered in the section "Setup environment" below.
-
-### Step 3. Create a Pull Request (PR)
-
-Once the change is ready, open a PR from your branch in your fork, to the `develop` branch in [securefederatedai/openfl](https://github.com/securefederatedai/openfl). OpenFL follows standard recommendations of PR formatting. Find more details [here](https://github.blog/2015-01-21-how-to-write-the-perfect-pull-request/).
-
-### Step 4. Sign your work
-
-Signoff your patch commits using your real name. We discourage anonymous contributions.
-
-    Signed-off-by: Joe Smith <[email protected]>
-
-If you set your `user.name` and `user.email` git configs, you can sign your
-commits using `git commit --signoff`.
-
-Your signature [certifies](http://developercertificate.org/) that you wrote the patch, or, you otherwise have the right to pass it on as an open-source patch.
-
-OpenFL is licensed under the [Apache 2.0 license](https://github.com/securefederatedai/openfl/blob/develop/LICENSE). By contributing to the project, you agree to the license and copyright terms therein and release your contribution under these terms.
-
-### Step 5. Code review and merge
-
-Verify that your contribution passes all tests in our CI/CD pipeline. In case of a failure, like shown below, look into the error messages and try to fix them.
-
-![CI/CD](docs/images/CI_details.png)
-
-Meanwhile, a reviewer will review the pull request and provide comments. Post few iterations of
-reviews and changes (depending on the complexity of the changes), PR will be approved for merge.
-
-## Setup environment
-
-We recommend setting up a local dev environment. Clone your forked repo to your local machine and install the dependencies.
-
-```shell
-git clone https://github.com/YOUR_GITHUB_USERNAME/openfl.git
-cd openfl
-pip install -U pip setuptools wheel
-pip install .
-pip install -r linters-requirements.txt
-```
-
-## Code style
-
-OpenFL uses [ruff](https://github.com/astral-sh/ruff) to lint/format code and [precommit](https://pre-commit.com/) checks.
-
-Run the following command at the **root** directory of the repo to format your code.
-
-```
-sh scripts/format.sh
-```
-You may need to resolve errors that could not be resolved by autoformatting. To only show lint errors, run `sh scripts/lint.sh` at the **root** directory of the repo.
-
-### Docstrings
-Since docstrings cannot be checked or standardized, if you do write/edit any docstring, make sure to check them manually. OpenFL docstrings should follow the conventions below:
-
-A **class** or a **function** docstring may contain:
-* A one-line description of the class/function.
-* Paragraph(s) of detailed information.
-* Optional `Examples` section.
-* `Args` section for arguments under `__init__()`.
+For more information, see [Contributing to OpenFL](https://openfl.readthedocs.io/en/latest/contributing.html).

ROADMAP.md

@@ -1,42 +1,2 @@
 # OpenFL Project Roadmap
-
-This document is intended to give users and contributors an idea of the OpenFL team's current priorities, features we plan to incorporate over the short, medium, and long term, and call out opportunities for the community to get involved.
-
-## When will this document be updated?
-At a minimum once each product release - which we expect to be on quarterly cadence.
-
-## 1. Features and interfaces
-
-### 1.1 Decoupling the FL specification interface from the infrastructure
-The task runner interface is coupled with the the single experiment aggregator / collaborator infrastructure, and the Interactive API is tied to the director / envoy infrastructure. 
-The Interactive API was originally designed to be a high-level API for OpenFL, but for the cases when more control is required by users, access to lower level interfaces is necessary.
-In OpenFL 1.5, we introduced the Workflow API as an experimental feature, which can be used to specify the federated learning flow, independently of the underlying computing infrastructure. The Workflow API facilitates a seamless transition from local simulation to a federated setting. Additionally, this approach offers greater control over the sequence and content of the FL experiment steps, which enables more complex experiments beyond just horizontal FL. Workflow API also provides more granular privacy controls, allowing the model owner to explicitly permit or forbid the transfer of specific attributes over the network.
-
-### 1.2 Consolidating interfaces
-OpenFL has supported multiple ways of running FL experiments for a long time, many of which are not interoperable: TaskRunner API, Workflow API, Python Native API, and Interactive API. The strategic vision is to consolidate OpenFL around the Workflow API, as it focuses on meeting the needs of the data scientist, who is the main user of the framework. Over the upcoming 1.x releases, we plan to gradually deprecate and eliminate the legacy Python Native API and Interactive API. OpenFL 2.0 will be centered around the Workflow API, facilitating a seamless transition from local simulations to distributed FL experiments, and even enabling the setup of permanent federations, which is currently only possible through the Interactive API.
-
-### 1.3 Component standardization and framework interoperability
-
-Federated Learning is a [burgeoning space](https://github.com/weimingwill/awesome-federated-learning#frameworks).
-Most core FL infrastructure (model weight extraction, network protocols, and serialization designs) must be reimplemented ad hoc by each framework. 
-This causes community fragmentation and distracts from some of the bigger problems to be solved in federated learning. In the short term, we want to collaborate on standards for FL,
- first at the communication and storage layer, and make these components modular across other frameworks. Our aim is also to provide a library for FL algorithms, compression methods,
- that can both be applied and interpreted easily.
-
- ### 1.4 Confidential computing support
- Although OpenFL currently relies on Intel® SGX for trusted execution, the long term vision is towards broader confidential computing ecosystem support. This can be achieved by packaging OpenFL workspaces and workflows as Confidential Containers (CoCo), which supports a spectrum of TEE backends, including Intel® SGX and TDX, Arm TrustZone, and AMD SEV.
-
-## Upcoming OpenFL releases
-
-### OpenFL 1.7 (Q1 2025)
-This release is focused on enabling a great developer experience for OpenFL users:
-1. Introducing the [FederatedRuntime](https://openfl.readthedocs.io/en/latest/about/features_index/workflowinterface.html#runtimes-future-plans) for Workflow API, which allows running FL workflows in a distributed setting (after local simulation with the LocalRuntime).
-2. Adding support for federated XGBoost in OpenFL. See the example [XGBoost workspace](https://github.com/securefederatedai/openfl/tree/develop/openfl-workspace/xgb_higgs) based on Task Runner API.
-3. Revised Task Runner API workspace dockerization process, with TEE-ready containers (using Gramine and Intel® Software Guard Extensions). The current release contains an initial set of changes that enable OpenFL compatibility with the broader confidential containers ecosystem.
-4. Streamlining the Federated Evaluation experiments with TaskRunner API
-5. Migrating a selection of key OpenFL tutorials from Python Native API to Workflow API. Check out the updated [Tutorials folder](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/experimental/workflow)
-6. Deprecating the Python Native API
-7. Deprecating the Interactive API
-
-### OpenFL 1.8 (TBA)
-Stay tuned for updates soon!
+For more information, see [roadmap](https://openfl.readthedocs.io/en/latest/roadmap.html).

docs/.gitignore

@@ -1,5 +1,3 @@
-# openfl*
-# models*
-# data*
 /_build
-**/.ipynb_checkpoints
+**/.ipynb_checkpoints
+_autosummary

docs/Makefile

@@ -1,13 +1,8 @@
-# Copyright (C) 2020-2023 Intel Corporation
-# SPDX-License-Identifier: Apache-2.0
-
-
 # Minimal makefile for Sphinx documentation
-#
 
 # You can set these variables from the command line, and also
 # from the environment for the first two.
-SPHINXOPTS    ?= -D autodoc_default_options.imported-members=True
+SPHINXOPTS    ?= -D autodoc_default_options.imported-members=True -D nb_execution_mode=force
 SPHINXBUILD   ?= sphinx-build
 SOURCEDIR     = .
 BUILDDIR      = _build
@@ -21,4 +16,4 @@ help:
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/README.md

@@ -1,29 +1,2 @@
-# How to update the documentation
-
-We use sphinx to generate the documentation for this project.
-The documentation project has been initialized properly and we basically just need to update the actual content.
-
-Install the openfl package
-
-```sh
-pip install .
-```
-
-Install requirements for building documentation:
-
-```sh
-pip install -r docs-requirements.txt
-```
-
-
-The Makefile supports many targets. We choose html because we can easily host the documentation on a remote server. Compile the documentation source code:
-```sh
-make clean
-make html
-```
-
-Open documentation locally:
-```sh
-cd _build/html
-python -m http.server
-```
+# Update documentation
+For more information, refer [here](https://openfl.readthedocs.io/en/latest/contributing.html#update-documentation).