examples/hello-world/step-by-step/higgs/stats/tabular_stats.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ccf1c433-ba7f-4967-b066-670eb5e016ed",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Tabular Data Federated Statistics \n",
    "\n",
    "Before we perform machine learning tasks on tabular data, it is often helpful to examine the statistics of the dataset on each client. This tutorial illustrates a federated statistics for tabular data. \n",
    "\n",
    "\n",
    "## Setup NVFLARE\n",
    "\n",
    "Follow [Getting Started](../../../../getting_started/readme.ipynb) to set up a virtual environment and install NVFLARE.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ebf2d68-b020-457f-a1f1-b1f95509c929",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Install requirements\n",
    "assuming the current directory is '/examples/hello-world/step-by-step/higgs/stats'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a552a5eb-dbfb-42da-8cfa-082c1739012e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f373c84-b3d9-43e2-9e6a-9ff0db33f3b4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ed5db39-b8f6-42bd-9610-6e671f17a6ea",
   "metadata": {},
   "source": [
    ">Note:\n",
    "In the upcoming sections, we'll utilize the 'tree' command. To install this command on a Linux system, you can use the sudo apt install tree command. As an alternative to 'tree', you can use the ls -al command."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3d5cd9a-3da9-446c-aac0-6ae84bf0ead1",
   "metadata": {},
   "source": [
    "\n",
    "## Prepare data\n",
    "Please reference [prepare_higgs_data](../prepare_data.ipynb) notebooks. Pay attention to the current location. You need to switch \"higgs\" directory to run the data split.\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dce2dd58-210f-4006-86ec-a821713d5cac",
   "metadata": {
    "tags": []
   },
   "source": [
    "Now we have our data prepared, let's first take a look at these data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47416d89-74f9-4ede-b8b9-99f0eeaa9814",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "features = [\"label\", \"lepton_pt\", \"lepton_eta\", \"lepton_phi\", \"missing_energy_magnitude\", \"missing_energy_phi\", \"jet_1_pt\", \"jet_1_eta\", \"jet_1_phi\", \"jet_1_b_tag\", \"jet_2_pt\", \"jet_2_eta\", \"jet_2_phi\", \"jet_2_b_tag\", \"jet_3_pt\", \"jet_3_eta\", \"jet_3_phi\", \"jet_3_b_tag\",\\\n",
    "            \"jet_4_pt\", \"jet_4_eta\", \"jet_4_phi\", \"jet_4_b_tag\", \\\n",
    "            \"m_jj\", \"m_jjj\", \"m_lv\", \"m_jlv\", \"m_bb\", \"m_wbb\", \"m_wwbb\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe4ab17c-8af7-43af-923d-becf6dec919c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d7ad635-5b8a-4430-8bce-8e3bb4b0a0d4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "df: pd.DataFrame = pd.read_csv(\"/tmp/nvflare/dataset/output/site-1.csv\", names=features, sep=r\"\\s*,\\s*\", engine=\"python\", na_values=\"?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1751d35c-99ce-4bef-9b8f-af3a57507a46",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f2cefa5-d579-45c0-815e-70dac06809c4",
   "metadata": {},
   "source": [
    "## Create a statistics calculator for the local tabular dataset\n",
    "\n",
    "We compose a calculator for getting the statistics of a tabular dataset, including count, sum, mean, stdev, etc. Read `./code/df_stats.py` for details\n",
    "\n",
    "Let's see if the code works. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c8e59bd-0b97-4ad9-a0de-abd10dd76d57",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cd code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "686d46c3-3986-4faf-bde8-1d4af749bf57",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from df_stats import DFStatistics\n",
    "\n",
    "df_stats_cal = DFStatistics(data_root_dir = \"/tmp/nvflare/dataset/output\")\n",
    "\n",
    "# We use fl_ctx = None for local calculation ( where the data set default to \"site-1.csv\", so we can explore the stats locally without federated settings. \n",
    "df_stats_cal.initialize(fl_ctx = None)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cef9f923-54eb-4b7b-a23d-c93201554332",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_features = df_stats_cal.features()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48c8f648-c5b9-4b71-b80c-431613c33dbc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0464f5e-b9ae-42d9-8bbe-7c180ce48767",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_stats_cal.count(\"train\", \"lepton_pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31419dd6-afdd-4612-becf-ce5a53d756cb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_stats_cal.mean(\"train\", \"lepton_pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36773f44-5153-4fcf-9814-c054f64f2723",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_stats_cal.mean(\"train\", \"m_wwbb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a5d0c62-1c7a-4a4c-bd98-3feb2c439268",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_stats_cal.stddev(\"train\", \"m_wwbb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c4599e3-bd78-4152-b255-246f801a51dc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_stats_cal.histogram(\"train\", \"lepton_pt\", 20, 0, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3735304-860d-41f1-bd9b-f05b1f048e4b",
   "metadata": {},
   "source": [
    "Great ! The code works. Let's move to the federated statistics calculations. Befor we do that, we need to move back to the parent directory of code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b0d8014-ed54-4d0e-bc96-adad905a29dd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cd ../."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d19adbde-b204-4483-810b-56c5d1517112",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Create Federated Statistics Job\n",
    "We are going to use Job API to construct a FedJob, then use it to run simulation or export job configs. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfe46054-ada9-41da-8879-373c78ed2431",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cat  code/df_stats_job.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e7b1fa7-1eef-42a4-b297-81289f4440c4",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Run job in FL Simulator\n",
    "\n",
    "Now we can run the job with simulator. There are two ways to run this. \n",
    "1) directly the job via job.simulator_run() \n",
    "2) generate job config, then use simulator CLI \n",
    " \n",
    "**Run job.simulator_run()**\n",
    "\n",
    "> note\n",
    "the data_root_dir=/tmp/nvflare/dataset/output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54c74444-58ab-4977-9d48-0e8dfb6f149d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! python code/df_stats_job.py -w /tmp/nvflare/tabular/stats_df -n 3 -d /tmp/nvflare/dataset/output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f10f0dc-28d1-469a-9977-e26fd0b9a06a",
   "metadata": {},
   "source": [
    "\n",
    "**Export job config, Run Job using Simulator CLI**\n",
    "\n",
    "```\n",
    "! python code/df_stats_job.py -co -j /tmp/nvflare/jobs/stats_df_job -n 3\n",
    "! nvflare simulator /tmp/nvflare/jobs/stats_df_job/stats_df/ -w /tmp/nvflare/tabular/stats_df -n 3 -t 3\n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9562e2d3-eb4a-4ee3-baa4-b9c1339a9954",
   "metadata": {},
   "source": [
    "\n",
    "### Examine Result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e981ae44-00b7-41b8-a2f0-4c49e05d6014",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "The results are stored in \n",
    "```\n",
    "/tmp/nvflare/tabular/stats_df/server/simulate_job/statistics/stats.json\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6117ca2-c9bf-43f1-98ed-4eb3dc8f369b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!ls -al /tmp/nvflare/tabular/stats_df/server/simulate_job/statistics/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "653f83f8-f96f-4943-af27-c5e6551d3449",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Result Visualization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef772576-d87a-4a6e-b530-6a440e230839",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "from nvflare.app_opt.statistics.visualization.statistics_visualization import Visualization\n",
    "with open('/tmp/nvflare/tabular/stats_df/server/simulate_job/statistics/stats.json', 'r') as f:\n",
    "    data = json.load(f)\n",
    "\n",
    "vis = Visualization()\n",
    "vis.show_stats(data = data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7cdfe52-28a4-499f-9884-d833fec6d3cb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from IPython.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:100%  depth:100% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47f354b9-6238-40be-91d9-67229c7b5891",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "vis.show_histograms(data = data, plot_type=\"main\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7a3d942-9b8d-45de-85aa-b21d27ca60ef",
   "metadata": {
    "tags": []
   },
   "source": [
    "Given the homogeneous data distribution across the 3 clients, the global histogram at each data point is relatively 3 times the local histograms. \n",
    "\n",
    "## We are done !\n",
    "Congratulations! you have just completed the federated stats calulation for tabular data. \n",
    "\n",
    "If you would like to see a detailed discussion regarding privacy filtering, please checkout the example in [federated statistics](https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/federated-statistics) examples.\n",
    "\n",
    "Let's move on to the next examples and see how can we use scikit-learn to train federated models on tabular data.\n",
    "First we will look at the [sklearn-linear](../sklearn-linear/sklearn_linear.ipynb) example, which illustrates how to train a federated linear model (logistic regression on binary classification)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba5e6273-5b79-4eb9-beb3-4ae96c21218c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nvflare_example",
   "language": "python",
   "name": "nvflare_example"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}