notebooks/classif_experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d23ef583",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#000000; border-radius: 0px; padding:25px;\"> Theoretical study of SMOTE </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "290db701",
   "metadata": {},
   "source": [
    "This notebook reproduces the numerical experiments.\n",
    "\n",
    "<ins>Experimented Datasets: </ins>\n",
    "- [Phoneme](#Phoneme)\n",
    "- Pima\n",
    "- Abalone\n",
    "- Haberman\n",
    "- Yeast\n",
    "- Wine\n",
    "- Vehicle\n",
    "- Bresat cancer Wisconsin\n",
    "- Ionosphere\n",
    "- CreditCard\n",
    "- MagicTel\n",
    "- California\n",
    "- House_16H\n",
    "\n",
    "ALl the sections of this notebook are indepent. user should run the following two introduction code chunk. Then one can choose a given data set section and run exclusivey the chunks code of the selected section. Fot the subsampled data sets version, all the seed and are given and the subsampling are each time included in the given section. However, the protocol is always pre-filed with the smalles (most undersampled) version of the dat set. On can try other subsampled data sets by changing the protocol data sets used.\n",
    "\n",
    "\n",
    "Tuning dpeth part is at the end of tghe notebook. The used depth for each data set are given. user should fill the protocol with the desired data sets and their associated depth dictionnary.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3b46d90",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "sys.path.insert(1, os.path.abspath(os.path.join(os.getcwd(), os.pardir)))\n",
    "from pathlib import Path\n",
    "\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier \n",
    "from sklearn.model_selection import GridSearchCV, StratifiedKFold\n",
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from imblearn.under_sampling import RandomUnderSampler,NearMiss\n",
    "from imblearn.over_sampling import RandomOverSampler, SMOTE, BorderlineSMOTE\n",
    "\n",
    "\n",
    "from oversampling_strategies.oversampling_strategies import MGS,MGS2, NoSampling, CVSmoteModel\n",
    "\n",
    "from validation.classif_experiments import run_eval, subsample_to_ratio_indices,read_subsampling_indices, compute_metrics_several_protocols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa869e32",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_dir_path =  \"../saved_experiments\" ## Fill it\n",
    "Path(output_dir_path).mkdir(parents=True, exist_ok=True)\n",
    "output_dir_path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45bf2347",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\" name='Phoneme'> Phoneme</h1></center>\n",
    "\n",
    "## <a name='Phoneme'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f1e2b43",
   "metadata": {},
   "source": [
    "## Run protocol :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33f7c3c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_phoneme_data\n",
    "X_phoneme, y_phoneme = load_phoneme_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3db2612e",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_kept_20 = subsample_to_ratio_indices(X=X_phoneme,y=y_phoneme,ratio=0.2,seed_sub=11,\n",
    "                                           output_dir_subsampling=output_dir_path,\n",
    "                                           name_subsampling_file='phoneme_sub_original_to_20')\n",
    "\n",
    "indices_kept_10 = subsample_to_ratio_indices(\n",
    "    X=X_phoneme,y=y_phoneme,ratio=0.1,seed_sub=9,\n",
    "    output_dir_subsampling=output_dir_path,\n",
    "    name_subsampling_file='phoneme_sub_20_to_10',has_previous_under_sampling=True,\n",
    "    previous_under_sampling=indices_kept_20)\n",
    "\n",
    "indices_kept_1 = subsample_to_ratio_indices(X=X_phoneme,y=y_phoneme,ratio=0.01,seed_sub=5,\n",
    "    output_dir_subsampling=output_dir_path,\n",
    "    name_subsampling_file='phoneme_sub_10_to_1',has_previous_under_sampling=True,\n",
    "    previous_under_sampling=indices_kept_10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "100e114b",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_phoneme_1,y_phoneme_1 = read_subsampling_indices(X=X_phoneme,y=y_phoneme, dir_subsampling=output_dir_path, name_subsampling_file='phoneme_sub_10_to_1', get_indexes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bfa0bac",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('MGS',MGS(K=6),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e636ca24",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_phoneme_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_phoneme_1,y=y_phoneme_1,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "364c49c0",
   "metadata": {},
   "source": [
    "## Compute metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0875b7f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4142c5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_phoneme,df_final_std_phoneme = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_phoneme_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "    \n",
    "\n",
    "df_final_mean_phoneme.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b64bf0a2",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> Pima </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab43b2e1",
   "metadata": {},
   "source": [
    "## Run protocol :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "934beb33",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_pima_data\n",
    "X_pima, y_pima = load_pima_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b5bfa68",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_kept_20 = subsample_to_ratio_indices(X=X_pima,y=y_pima,ratio=0.2,seed_sub=15,\n",
    "                                           output_dir_subsampling=output_dir_path,\n",
    "                                           name_subsampling_file='pima_sub_original_to_20')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3cb5126",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_pima_20,y_pima_20 = read_subsampling_indices(X=X_pima,y=y_pima, dir_subsampling=output_dir_path, name_subsampling_file='pima_sub_original_to_20', get_indexes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "171f6ed2",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=9,n_points=9,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a653b72",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_pima_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_pima_20,y=y_pima_20,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57dd83eb",
   "metadata": {},
   "source": [
    "## Compute metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ded77f18",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "544c9de6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_pima,df_final_std_pima = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_pima_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_pima.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25bcd19d",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> Abalone </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28556d4e",
   "metadata": {},
   "source": [
    "## Run protocol :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18611926",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_abalone_data\n",
    "X_abalone, y_abalone = load_abalone_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8742c41f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "Counter(y_abalone)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecd023ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=9,n_points=9,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89e72d3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_abalone_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_abalone,y=y_abalone,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "befbf814",
   "metadata": {},
   "source": [
    "## Compute metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8bf58ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0f9fbc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_abalone,df_final_std_abalone = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_abalone_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_abalone.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc84a06c",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> Haberman </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "112be115",
   "metadata": {},
   "source": [
    "## Run protocol :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8d9171a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_haberman_data\n",
    "X_haberman,y_haberman = load_haberman_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2178b0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_kept_10 = subsample_to_ratio_indices(X=X_haberman,y=y_haberman,ratio=0.1,seed_sub=11,\n",
    "                                           output_dir_subsampling=output_dir_path,\n",
    "                                           name_subsampling_file='haberman_sub_original_to_20')\n",
    "X_haberman_10,y_haberman_10 = read_subsampling_indices(X=X_haberman,y=y_haberman, dir_subsampling=output_dir_path, name_subsampling_file='haberman_sub_original_to_20', get_indexes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f238c57a",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=5,n_points=5,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0abf5ae7",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_haberman_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_haberman_10,y=y_haberman_10,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43f6a16d",
   "metadata": {},
   "source": [
    "## Compute metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c3198d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from validation.classif_experiments import compute_metrics_several_protocols\n",
    "\n",
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50005730",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_haberman,df_final_std_haberman = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_haberman_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_haberman.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2211351d",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> Yeast </h1></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e39fe093",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_yeast_data\n",
    "X_yeast,y_yeast = load_yeast_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "271cc234",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_kept_10 = subsample_to_ratio_indices(X=X_yeast,y=y_yeast,ratio=0.01,seed_sub=15,\n",
    "                                           output_dir_subsampling=output_dir_path,\n",
    "                                           name_subsampling_file='yeast_sub_original_to_1')\n",
    "\n",
    "X_yeast_1,y_yeast_1 = read_subsampling_indices(X=X_yeast,y=y_yeast, dir_subsampling=output_dir_path, name_subsampling_file='yeast_sub_original_to_1', get_indexes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7744886e",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=7,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=9,n_points=9,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e98a7e95",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_yeast_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_yeast_1,y=y_yeast_1,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeca2f81",
   "metadata": {},
   "source": [
    "## Compute Metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6005f90e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from validation.classif_experiments import compute_metrics_several_protocols\n",
    "\n",
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caf68e3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_haberman,df_final_std_haberman = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_yeast_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_haberman.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "771b0425",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> Wine </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86e50d65",
   "metadata": {},
   "source": [
    "Wine quality datas et : class 5 and 6 vs 8. Do not forget to install ucimlrepo. # pip install ucimlrepo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7896550a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_wine_data\n",
    "X_wine,y_wine = load_wine_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9612ba62",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=12,n_points=12,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33fc2d9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_wine_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_wine,y=y_wine,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56dcaff2",
   "metadata": {},
   "source": [
    "## Compute Metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92ce81fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from validation.classif_experiments import compute_metrics_several_protocols\n",
    "\n",
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f208a8b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_haberman,df_final_std_haberman = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_wine_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_haberman.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24d73a14",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> Vehicle </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d19efa62",
   "metadata": {},
   "source": [
    "## Run protocol :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aabdaf52",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_vehicle_data\n",
    "X_vehicle, y_vehicle = load_vehicle_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8e833db",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_kept_10 = subsample_to_ratio_indices(X=X_vehicle,y=y_vehicle,ratio=0.1,seed_sub=11,\n",
    "    output_dir_subsampling=output_dir_path,\n",
    "    name_subsampling_file='vehicle_sub_original_to_10')\n",
    "\n",
    "X_vehicle_10, y_vehicle_10 = read_subsampling_indices(X=X_vehicle,y=y_vehicle, dir_subsampling=output_dir_path, name_subsampling_file='vehicle_sub_original_to_10', get_indexes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "376731bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=19,n_points=19,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0f6cc03",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_vehicle_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_vehicle_10,y=y_vehicle_10,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae4c1c26",
   "metadata": {},
   "source": [
    "## Compute Metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6999e72e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from validation.classif_experiments import compute_metrics_several_protocols\n",
    "\n",
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccb28aa2",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_vehicle,df_final_std_vehicle = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_vehicle_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_vehicle.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6bdb8b5",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> Breast cancer Wisconsin </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59c0dc80",
   "metadata": {},
   "source": [
    "## Run protocol :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94a6e3a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_breastcancer_data\n",
    "X_bcw, y_bcw = load_breastcancer_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f35f5172",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_kept_20 = subsample_to_ratio_indices(X=X_bcw,y=y_bcw,ratio=0.2,seed_sub=11,\n",
    "                                           output_dir_subsampling=output_dir_path,\n",
    "                                           name_subsampling_file='bcw_sub_original_to_20')\n",
    "\n",
    "indices_kept_10 = subsample_to_ratio_indices(\n",
    "    X=X_bcw,y=y_bcw,ratio=0.1,seed_sub=9,\n",
    "    output_dir_subsampling=output_dir_path,\n",
    "    name_subsampling_file='bcw_sub_20_to_10',has_previous_under_sampling=True,\n",
    "    previous_under_sampling=indices_kept_20)\n",
    "\n",
    "X_bcw_10, y_bcw_10 = read_subsampling_indices(X=X_bcw,y=y_bcw, dir_subsampling=output_dir_path, name_subsampling_file='bcw_sub_20_to_10', get_indexes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d94f37c",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=10,n_points=10,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f542208",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_bcw_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_bcw_10,y=y_bcw_10,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f30beae",
   "metadata": {},
   "source": [
    "## Compute Metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f96575f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from validation.classif_experiments import compute_metrics_several_protocols\n",
    "\n",
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6cc70bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_haberman,df_final_std_haberman = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_bcw_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_haberman.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db3498d7",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> Ionosphere </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee59139b",
   "metadata": {},
   "source": [
    "## Run protocol :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e9bbdbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_ionosphere_data\n",
    "X_ionosphere, y_ionosphere = load_ionosphere_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15c33da0",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_kept_20 = subsample_to_ratio_indices(X=X_ionosphere,y=y_ionosphere,ratio=0.2,seed_sub=11,\n",
    "                                           output_dir_subsampling=output_dir_path,\n",
    "                                           name_subsampling_file='ionosphere_sub_original_to_20')\n",
    "\n",
    "indices_kept_10 = subsample_to_ratio_indices(\n",
    "    X=X_ionosphere,y=y_ionosphere,ratio=0.1,seed_sub=9,\n",
    "    output_dir_subsampling=output_dir_path,\n",
    "    name_subsampling_file='ionosphere_sub_20_to_10',has_previous_under_sampling=True,\n",
    "    previous_under_sampling=indices_kept_20)\n",
    "\n",
    "X_ionosphere_10, y_ionosphere_10 = read_subsampling_indices(X=X_ionosphere,y=y_ionosphere, dir_subsampling=output_dir_path, name_subsampling_file='ionosphere_sub_20_to_10', get_indexes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70523aab",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=5,n_points=5,llambda=1.0),{},rf_model), # Not possible for Inosphere 10% to have MGS K=33 (wo we take K=5). Inosphere and Ionopshere 20% were performed with K=33, change it if they are used\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a027c8c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_ionosphere_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_ionosphere_10,y=y_ionosphere_10,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caebe2d5",
   "metadata": {},
   "source": [
    "## Compute Metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b7dbf24",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from validation.classif_experiments import compute_metrics_several_protocols\n",
    "\n",
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a22c849",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_haberman,df_final_std_haberman = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_ionosphere_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_haberman.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "729864b3",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> CreditCard </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41804b7e",
   "metadata": {},
   "source": [
    "## Run protocol :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f323f1b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_credit_data\n",
    "X_credit, y_credit, meta_df_credit = load_credit_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa4047c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from validation.classif_experiments import PaperTimeSeriesSplitWithGroupOut\n",
    "splitter_PTSSG = PaperTimeSeriesSplitWithGroupOut(n_splits=5, starting_split=0,meta_df=meta_df_credit,col_name_id='Time')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7641df7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=30,n_points=30,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0577d86",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_credit_depthNone_test'\n",
    "for i in range(1):\n",
    "# We do not change the splitter at each iteration here due to the temporal split\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_credit,y=y_credit,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_PTSSG)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "078f1253",
   "metadata": {},
   "source": [
    "## Compute Metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfb819d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from validation.classif_experiments import compute_metrics_several_protocols\n",
    "\n",
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94d61dfc",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_credit,df_final_std_credit = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_credit_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_credit.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8360608c",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> MagicTel </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efe54ebd",
   "metadata": {},
   "source": [
    "## Run protocol : \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13dfc6f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_magictel_data\n",
    "X_magic,y_magic = load_magictel_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89b3b1f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_kept_20 = subsample_to_ratio_indices(X=X_magic,y=y_magic,ratio=0.2,seed_sub=11,\n",
    "                                           output_dir_subsampling=output_dir_path,\n",
    "                                           name_subsampling_file='magicTel_sub_original_to_20')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d05af15",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_magic_20,y_magic_20 = read_subsampling_indices(X=X_magic,y=y_magic, dir_subsampling=output_dir_path, name_subsampling_file='magicTel_sub_original_to_20', get_indexes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5536aff",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=100)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=11,n_points=11,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f6a4e12",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_magicTel_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_magic_20,y=y_magic_20,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "107bd10e",
   "metadata": {},
   "source": [
    "## Compute Metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8ba7bf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from validation.classif_experiments import compute_metrics_several_protocols\n",
    "\n",
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8869e89a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_haberman,df_final_std_haberman = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_magicTel_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_haberman.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fab53a5d",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> California </h1></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88f5f463",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_california_data\n",
    "X_california,y_california = load_california_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9b9d153",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_kept_20 = subsample_to_ratio_indices(X=X_california,y=y_california,ratio=0.2,seed_sub=11,\n",
    "                                           output_dir_subsampling=output_dir_path,\n",
    "                                           name_subsampling_file='california_sub_original_to_20')\n",
    "\n",
    "indices_kept_10 = subsample_to_ratio_indices(\n",
    "    X=X_california,y=y_california,ratio=0.1,seed_sub=9,\n",
    "    output_dir_subsampling=output_dir_path,\n",
    "    name_subsampling_file='california_sub_20_to_10',has_previous_under_sampling=True,\n",
    "    previous_under_sampling=indices_kept_20)\n",
    "\n",
    "indices_kept_1 = subsample_to_ratio_indices(X=X_california,y=y_california,ratio=0.01,seed_sub=5,\n",
    "    output_dir_subsampling=output_dir_path,\n",
    "    name_subsampling_file='california_sub_10_to_1',has_previous_under_sampling=True,\n",
    "    previous_under_sampling=indices_kept_10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de5d5816",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_california_1,y_california_1 = read_subsampling_indices(X=X_california,y=y_california, dir_subsampling=output_dir_path, name_subsampling_file='california_sub_10_to_1', get_indexes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe0a365d",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=9,n_points=9,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c4c9cb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_california_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_california_1,y=y_california_1,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bc1ddcc",
   "metadata": {},
   "source": [
    "## Compute Metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94c398a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from validation.classif_experiments import compute_metrics_several_protocols\n",
    "\n",
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "896a5bdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_haberman,df_final_std_haberman = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_california_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_haberman.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46cfac40",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> House_16H </h1></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d25cd84",
   "metadata": {},
   "outputs": [],
   "source": [
    "from data.data import load_house_data\n",
    "X_house,y_house = load_house_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b132ecc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_kept_20 = subsample_to_ratio_indices(X=X_house,y=y_house,ratio=0.2,seed_sub=11,\n",
    "                                           output_dir_subsampling=output_dir_path,\n",
    "                                           name_subsampling_file='house16_sub_original_to_20')\n",
    "\n",
    "indices_kept_10 = subsample_to_ratio_indices(\n",
    "    X=X_house,y=y_house,ratio=0.1,seed_sub=9,\n",
    "    output_dir_subsampling=output_dir_path,\n",
    "    name_subsampling_file='house16_sub_20_to_10',has_previous_under_sampling=True,\n",
    "    previous_under_sampling=indices_kept_20)\n",
    "\n",
    "indices_kept_1 = subsample_to_ratio_indices(X=X_house,y=y_house,ratio=0.01,seed_sub=5,\n",
    "    output_dir_subsampling=output_dir_path,\n",
    "    name_subsampling_file='house16_sub_10_to_1',has_previous_under_sampling=True,\n",
    "    previous_under_sampling=indices_kept_10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00497731",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_house_1,y_house_1 = read_subsampling_indices(X=X_house,y=y_house, dir_subsampling=output_dir_path, name_subsampling_file='house16_sub_10_to_1', get_indexes=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b580c82",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5)\n",
    "balanced_rf_model = RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=5)\n",
    "\n",
    "splitter_stratified_cv = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "CVSmoteRF = CVSmoteModel(\n",
    "    splitter=splitter_stratified_cv,\n",
    "    model=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=5),\n",
    "    list_k_max=15,list_k_step=1)\n",
    "\n",
    "list_oversampling_and_params = [\n",
    "    ('None', NoSampling(), {}, rf_model),\n",
    "    ('c_weight', NoSampling(), {}, balanced_rf_model),\n",
    "    ('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, rf_model),\n",
    "    ('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, rf_model),\n",
    "    ('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, rf_model),\n",
    "    ('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, rf_model),\n",
    "    ('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, rf_model),\n",
    "    ('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},rf_model),\n",
    "    ('cv_smote', NoSampling(),{}, CVSmoteRF),\n",
    "    ('MGS',MGS(K=17,n_points=17,llambda=1.0),{},rf_model),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3eeca6a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "init_name_file = '2024-02-14-RF100_house16_depthNone_test'\n",
    "for i in range(100):\n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "    name_file = init_name_file + str(i) +'.npy'\n",
    "    run_eval(output_dir=output_dir_path,name_file=name_file,X=X_house_1,y=y_house_1,\n",
    "             list_oversampling_and_params=list_oversampling_and_params,\n",
    "             splitter=splitter_stratified)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a75cae0",
   "metadata": {},
   "source": [
    "## Compute Metrics :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1e49352",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, roc_auc_score\n",
    "from validation.classif_experiments import compute_metrics_several_protocols\n",
    "\n",
    "list_metric = [\n",
    "    (precision_score,'precision','pred'),\n",
    "    (recall_score,'recall','pred'),\n",
    "    (roc_auc_score,'roc_auc','proba')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e41be3de",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final_mean_haberman,df_final_std_haberman = compute_metrics_several_protocols(\n",
    "    output_dir=output_dir_path,\n",
    "    init_name_file='2024-02-14-RF100_house16_depthNone_test',\n",
    "    list_metric=list_metric,\n",
    "    bool_roc_auc_only=True,n_iter=100)\n",
    "\n",
    "df_final_mean_haberman.style.format(precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d1d83d3",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#808b96; border-radius: 10px; padding:15px;\"> Tuning depth </h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1c51fc7",
   "metadata": {},
   "source": [
    "For tunnig depth, for each data sets, we look at the mean depth of the forests for a given strategy (computed when the protocol is running and saved in a file). Then we applied a gridsearch with a grid for max_depth parameter based on the maximum value previously selected.\n",
    "The maximum value dictionnary of depth for ecah highly imbalanced dat are given below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa223f8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from validation.distances_experiments import depth_func_linspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e00f0118",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_pima_20 = {'none':13,\n",
    "                           'smote':15,\n",
    "                           'MGS':15,\n",
    "                           'cweight':14,\n",
    "                           'BS1':15,\n",
    "                           'BS2':16,\n",
    "                           'NM1':11,\n",
    "                           'RUS':10,\n",
    "                           'ROS':14,\n",
    "                           'cvsmote':15,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f001db98",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_abalone = {'none': 13,\n",
    "                           'smote':22,\n",
    "                           'MGS':22,\n",
    "                           'cweight':18,\n",
    "                           'BS1': 20,\n",
    "                           'BS2': 21,\n",
    "                           'NM1':7,\n",
    "                           'RUS':7,\n",
    "                           'ROS':18,\n",
    "                           'cvsmote':23,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16ba8dac",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_phoneme_1 = {'none': 11,\n",
    "                           'smote':20,\n",
    "                           'MGS':19,\n",
    "                           'cweight':15,\n",
    "                           'BS1': 16,\n",
    "                           'BS2': 19,\n",
    "                           'NM1':6,\n",
    "                           'RUS':6,\n",
    "                           'ROS':17,\n",
    "                           'cvsmote':21,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9022b969",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_yeast_1 = {'none': 9,\n",
    "                           'smote':14,\n",
    "                           'MGS':10,\n",
    "                           'cweight':11,\n",
    "                           'BS1': 13,\n",
    "                           'BS2': 15,\n",
    "                           'NM1':5,\n",
    "                           'RUS':4,\n",
    "                           'ROS':13,\n",
    "                           'cvsmote':14,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6903bbda",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_haberman_10 = {'none': 10,\n",
    "                           'smote':14,\n",
    "                           'MGS':14,\n",
    "                           'cweight':10,\n",
    "                           'BS1': 13,\n",
    "                           'BS2': 13,\n",
    "                           'NM1':7,\n",
    "                           'RUS':7,\n",
    "                           'ROS':11,\n",
    "                           'cvsmote':13,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dda9c241",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_wine = {'none': 16,\n",
    "                           'smote':25,\n",
    "                           'MGS':23,\n",
    "                           'cweight':24,\n",
    "                           'BS1': 25,\n",
    "                           'BS2': 26,\n",
    "                           'NM1':14,\n",
    "                           'RUS':10,\n",
    "                           'ROS':24,\n",
    "                           'cvsmote':26,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69db83b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_magic_20 = {'none':23,\n",
    "                           'smote':30,\n",
    "                           'MGS':26,\n",
    "                           'cweight':29,\n",
    "                           'BS1':31,\n",
    "                           'BS2':33,\n",
    "                           'NM1':21,\n",
    "                           'RUS':20,\n",
    "                           'ROS':29,\n",
    "                           'cvsmote':30,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "554f2ae2",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_creditcard = {'none': 18,\n",
    "                 'smote': 29,\n",
    "                 'MGS':36,\n",
    "                 'cweight':20,\n",
    "                 'BS1': 23,\n",
    "                 'BS2':29,\n",
    "                 'NM1':8,\n",
    "                 'RUS':10,\n",
    "                 'ROS':22,\n",
    "                 'cvsmote':29,}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39634967",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_california_1 = {'none':17,\n",
    "                           'smote':26,\n",
    "                           'MGS':26,\n",
    "                           'cweight':21,\n",
    "                           'BS1':23,\n",
    "                           'BS2':25,\n",
    "                           'NM1':9,\n",
    "                           'RUS':10,\n",
    "                           'ROS':24,\n",
    "                           'cvsmote':27,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bb8af22",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_vehicle_original = {'none':10,\n",
    "                                            'smote':12,\n",
    "                                            'MGS':11,\n",
    "                                            'cweight':12,\n",
    "                                            'BS1':12,\n",
    "                                            'BS2':12,\n",
    "                                            'NM1':9,\n",
    "                                            'RUS':9,\n",
    "                                            'ROS':12,\n",
    "                                            'cvsmote':12,\n",
    "                                        }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54ac158a",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_vehicle_10 = {'none':9,\n",
    "                           'smote':11,\n",
    "                           'MGS':11,\n",
    "                           'cweight':11,\n",
    "                           'BS1':11,\n",
    "                           'BS2':11,\n",
    "                           'NM1':7,\n",
    "                           'RUS':6,\n",
    "                           'ROS':12,\n",
    "                           'cvsmote':11,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdb45d64",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_bcw_original = {'none':8,\n",
    "                           'smote':8,\n",
    "                           'MGS':8,\n",
    "                           'cweight':8,\n",
    "                           'BS1':9,\n",
    "                           'BS2':9,\n",
    "                           'NM1':7,\n",
    "                           'RUS':7,\n",
    "                           'ROS':8,\n",
    "                           'cvsmote':8,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "808d0406",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_bcw_20 = {'none':6,\n",
    "                           'smote':7,\n",
    "                           'MGS':7,\n",
    "                           'cweight':7,\n",
    "                           'BS1':7,\n",
    "                           'BS2':8,\n",
    "                           'NM1':6,\n",
    "                           'RUS':4,\n",
    "                           'ROS':7,\n",
    "                           'cvsmote':7,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "366d6b7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_bcw_10 = {'none':4,\n",
    "                           'smote':6,\n",
    "                           'MGS':6,\n",
    "                           'cweight':6,\n",
    "                           'BS1':7,\n",
    "                           'BS2':7,\n",
    "                           'NM1':4,\n",
    "                           'RUS':3,\n",
    "                           'ROS':7,\n",
    "                           'cvsmote':6,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fbfdc9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_ionosphere_original = {'none':10,\n",
    "                           'smote':10,\n",
    "                           'MGS':10,\n",
    "                           'cweight':9,\n",
    "                           'BS1':10,\n",
    "                           'BS2':11,\n",
    "                           'NM1':7,\n",
    "                           'RUS':8,\n",
    "                           'ROS':9,\n",
    "                           'cvsmote':10,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea3179ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_ionosphere_20 = {'none':7,\n",
    "                           'smote':8,\n",
    "                           'MGS':8,\n",
    "                           'cweight':6,\n",
    "                           'BS1':7,\n",
    "                           'BS2':8,\n",
    "                           'NM1':4,\n",
    "                           'RUS':4,\n",
    "                           'ROS':7,\n",
    "                           'cvsmote':8,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6a18dc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat_ionosphere_10 = {'none':4,\n",
    "                           'smote':7,\n",
    "                           'MGS':7,\n",
    "                           'cweight':4,\n",
    "                           'BS1':6,\n",
    "                           'BS2':7,\n",
    "                           'NM1':3,\n",
    "                           'RUS':3,\n",
    "                           'ROS':6,\n",
    "                           'cvsmote':7,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14ede657",
   "metadata": {},
   "source": [
    "User should selec a higly imbalanced data set max depth dictionnary. Then, change the K value of MGS (K=dimension of the selected data set). Finnaly fill the data in run_eval function before launching the protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6547d3c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth_dict_by_strat = #Filll it\n",
    "\n",
    "for strat in max_depth_dict_by_strat:\n",
    "    max_depth = max_depth_dict_by_strat[strat]\n",
    "    list_depth = depth_func_linspace(min_value=5,max_value=max_depth,size=8)\n",
    "    ############\n",
    "    GRF = GridSearchCV(estimator=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=10),\n",
    "                       param_grid={'max_depth': list_depth }, scoring='roc_auc')\n",
    "    GRF2 = GridSearchCV(estimator=RandomForestClassifier(n_estimators=100,criterion='gini',class_weight='balanced',n_jobs=10),\n",
    "                        param_grid={'max_depth': list_depth }, scoring='roc_auc')\n",
    "    \n",
    "    splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=None)\n",
    "    CVSmoteGRF = CVSmoteModel(splitter=splitter_stratified,\n",
    "                             model=GridSearchCV(estimator=RandomForestClassifier(n_estimators=100,criterion='gini',n_jobs=10),\n",
    "                                                param_grid={'max_depth': list_depth}, scoring='roc_auc'),\n",
    "                                                list_k_max=15,list_k_step=1)\n",
    "    \n",
    "    list_oversampling_dict_by_strat = {'none': [('None', NoSampling(), {}, GRF),],\n",
    "                                       'smote': [('SMOTE',SMOTE(sampling_strategy=\"minority\"),{},GRF),],\n",
    "                                       'MGS':[('MGS',MGS(K=9,n_points=9,llambda=1.0),{},GRF)], # Adapt the dimension : K=n_points=dimesnion+1\n",
    "                                       'cweight':[('c_weight', NoSampling(), {}, GRF2),],\n",
    "                                       'BS1': [('BorderlineS_1', BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-1\"),{}, GRF),],\n",
    "                                       'BS2': [('BorderlineS_2',BorderlineSMOTE(sampling_strategy=\"minority\", k_neighbors=5, kind=\"borderline-2\"),{}, GRF),],\n",
    "                                       'NM1':[('NearMiss1', NearMiss(sampling_strategy=\"majority\", version=1),{}, GRF),],\n",
    "                                       'RUS':[('RUS', RandomUnderSampler(sampling_strategy=\"majority\",replacement=False),{}, GRF),],\n",
    "                                       'ROS':[('ROS',  RandomOverSampler(sampling_strategy=\"minority\"),{}, GRF),],\n",
    "                                       'cvsmote': [('cv_smote', NoSampling(),{}, CVSmoteGRF),]}\n",
    "    list_oversampling_and_params = list_oversampling_dict_by_strat[strat]\n",
    "    #############\n",
    "    output_dir=os.path.join(output_dir_path,strat,'RF_100')\n",
    "    Path(output_dir).mkdir(parents=True, exist_ok=True)\n",
    "    init_name_file = '2024-02-14-RF100_tuned_depthNone_'\n",
    "    for i in range(100):\n",
    "        splitter_stratified = StratifiedKFold(n_splits=5,shuffle=True,random_state=100+i)\n",
    "        name_file = init_name_file + str(i) +'.npy'\n",
    "        run_eval(output_dir=output_dir,name_file=name_file,X=,y=, # Fill data\n",
    "                 list_oversampling_and_params=list_oversampling_and_params,\n",
    "                 splitter=splitter_stratified,to_standard_scale=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1fed225",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_name_strat = ['none','cweight','RUS','ROS','NM1','BS1','BS2','smote','cvsmote','MGS']\n",
    "for strat in list_name_strat:\n",
    "    output_dir=output_dir=os.path.join(output_dir_path,strat,'RF_100')\n",
    "    init_name_file = '2024-02-14-RF100_tuned_depthNone_'\n",
    "\n",
    "    df_final_mean_abalone,df_final_std_abalone = compute_metrics_several_protocols(output_dir=output_dir,init_name_file=init_name_file,\n",
    "                                                                                   list_metric=list_metric,bool_roc_auc_only=False,n_iter=20)\n",
    "    mean_roc = round(df_final_mean_abalone.to_numpy()[0,0],3)\n",
    "    srtd_roc = round(df_final_std_abalone.to_numpy()[0,0],3)\n",
    "    print(\"########\"*5)\n",
    "    print('****** Strategy : ',strat, '******')\n",
    "    print(\"ROC AUC mean : \",mean_roc)\n",
    "    print(\"ROC AUC std : \",srtd_roc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cffa5f41",
   "metadata": {},
   "source": [
    "<center><h1  style=\"color:white; background-color:#008b96; border-radius: 10px; padding:15px;\"> END </h1></center>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}