SYNOP/SYNOP3-12h-predictions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n",
    "\n",
    "# <!-- TITLE --> [SYNOP3] - 12h predictions\n",
    "<!-- DESC --> Episode 3: Attempt to predict in a more longer term \n",
    "<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->\n",
    "\n",
    "## Objectives :\n",
    " - Prediction at 12:00\n",
    " - Understand the principle of using recurrent neurons... and the limitations of our example !\n",
    "\n",
    "\n",
    "SYNOP meteorological data, available at: https://public.opendatasoft.com\n",
    "\n",
    "## What we're going to do :\n",
    "\n",
    " - Read the data\n",
    " - Make a reccurent prediction\n",
    "\n",
    "## Step 1 - Import and init\n",
    "### 1.1 - Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras.callbacks import TensorBoard\n",
    "from tensorflow.keras.preprocessing.sequence import TimeseriesGenerator\n",
    "\n",
    "import numpy as np\n",
    "import math, random\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pandas as pd\n",
    "import h5py, json\n",
    "import os,time,sys\n",
    "\n",
    "from importlib import reload\n",
    "\n",
    "sys.path.append('..')\n",
    "import fidle.pwk as pwk\n",
    "\n",
    "run_dir = './run/SYNOP'\n",
    "datasets_dir = pwk.init('SYNOP3', run_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 - Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ---- About dataset (no need to change)\n",
    "#\n",
    "dataset_dir      = './data'                  # Enhanced dataset is very small, so ./data in a good choice :-)\n",
    "dataset_filename = 'synop-LYS.csv'\n",
    "schema_filename  = 'synop.json'\n",
    "features         = ['tend', 'cod_tend', 'dd', 'ff', 'td', 'u', 'ww', 'pres', 'rafper', 'rr1', 'rr3', 'tc']\n",
    "features_len     = len(features)\n",
    "\n",
    "# ---- About training\n",
    "#\n",
    "iterations       = 4        # number of iterations for prediction (1 iteration = 3h)\n",
    "\n",
    "scale            = 1        # Percentage of dataset to be used (1=all)\n",
    "train_prop       = .8       # Percentage for train (the rest being for the test)\n",
    "sequence_len     = 16       # Sequence len"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Override parameters (batch mode) - Just forget this cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwk.override('iterations', 'scale', 'train_prop', 'sequence_len')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Read and prepare dataset\n",
    "**Note** : The `scale` and `train_prop` parameters must be identical to those used during training (SYNOP2)... ;-)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ---- Read dataset\n",
    "\n",
    "df = pd.read_csv(f'{dataset_dir}/{dataset_filename}', header=0, sep=';')\n",
    "\n",
    "# ---- Scaling\n",
    "\n",
    "df = df[:int(scale*len(df))]\n",
    "train_len=int(train_prop*len(df))\n",
    "\n",
    "# ---- Train / Test\n",
    "dataset_train = df.loc[ :train_len-1, features ]\n",
    "dataset_test  = df.loc[train_len:,    features ]\n",
    "\n",
    "# ---- Normalize, and convert to numpy array\n",
    "mean = dataset_train.mean()\n",
    "std  = dataset_train.std()\n",
    "dataset_train = np.array( (dataset_train - mean) / std )\n",
    "dataset_test  = np.array( (dataset_test  - mean) / std )\n",
    "\n",
    "print('Dataset       : ',df.shape)\n",
    "print('Train dataset : ',dataset_train.shape)\n",
    "print('Test  dataset : ',dataset_test.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 - Load model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_model = tf.keras.models.load_model(f'{run_dir}/best_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Make a 12h prediction\n",
    "Note : Our predictions are normalized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# ---- Initial sequence\n",
    "\n",
    "s=random.randint(0,len(dataset_test)-sequence_len-iterations)\n",
    "\n",
    "sequence_pred = dataset_test[s:s+sequence_len].copy()\n",
    "sequence_true = dataset_test[s:s+sequence_len+iterations].copy()\n",
    "\n",
    "# ---- Iterate on 4 predictions\n",
    "\n",
    "sequence_pred=list(sequence_pred)\n",
    "\n",
    "for i in range(iterations):\n",
    "    sequence=sequence_pred[-sequence_len:]\n",
    "    pred = loaded_model.predict( np.array([sequence]) )\n",
    "    sequence_pred.append(pred[0])\n",
    "\n",
    "# ---- Extract the predictions    \n",
    "\n",
    "pred=np.array(sequence_pred[-iterations:])\n",
    "       \n",
    "# ---- Show result\n",
    "\n",
    "pwk.plot_multivariate_serie(sequence_true, predictions=pred, labels=features, save_as='01-prediction-norm')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Full prediction\n",
    "#### Some cool functions that do the job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def denormalize(mean,std,seq):\n",
    "    nseq = seq.copy()\n",
    "    for i,s in enumerate(nseq):\n",
    "        s = s*std + mean\n",
    "        nseq[i]=s\n",
    "    return nseq\n",
    "\n",
    "\n",
    "def get_prediction(dataset, model, iterations=4,sequence_len=16):\n",
    "\n",
    "    # ---- Initial sequence\n",
    "\n",
    "    s=random.randint(0,len(dataset)-sequence_len-iterations)\n",
    "\n",
    "    sequence_pred = dataset[s:s+sequence_len].copy()\n",
    "    sequence_true = dataset[s:s+sequence_len+iterations].copy()\n",
    "\n",
    "    # ---- Iterate\n",
    "\n",
    "    sequence_pred=list(sequence_pred)\n",
    "\n",
    "    for i in range(iterations):\n",
    "        sequence=sequence_pred[-sequence_len:]\n",
    "        pred = model.predict( np.array([sequence]) )\n",
    "        sequence_pred.append(pred[0])\n",
    "\n",
    "    # ---- Extract the predictions    \n",
    "\n",
    "    pred=np.array(sequence_pred[-iterations:])\n",
    "\n",
    "    # ---- De-normalization\n",
    "\n",
    "    sequence_true = denormalize(mean,std, sequence_true)\n",
    "    pred          = denormalize(mean,std, pred)\n",
    "\n",
    "    return sequence_true,pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### And the result is..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "  \n",
    "sequence_true, pred = get_prediction(dataset_test, loaded_model,iterations=4)\n",
    "\n",
    "feat=11\n",
    "\n",
    "pwk.plot_multivariate_serie(sequence_true, predictions=pred, labels=features,\n",
    "                            only_features=[feat],width=14, height=8, save_as='02-prediction')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwk.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"todo\">\n",
    "    What you can do:\n",
    "    <ul>\n",
    "        <li>Trying to increase the forecasting time</li>\n",
    "        <li>What could we do to try to improve our forecasts?</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<img width=\"80px\" src=\"../fidle/img/00-Fidle-logo-01.svg\"></img>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}