hackathon_2020.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict Power Generation of Solar Panels\n",
    "### GMiS CAHSI Data Analytics Hackathon 2020 - Team 14: Mask-araid\n",
    "----\n",
    "__Question__: What will be the generated power voltage from a solar panel at a given time in the future given the weather conditions?\n",
    "\n",
    "__Software Used__: Jupyter Lab\n",
    "\n",
    "__Programming Language__: Python "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Research\n",
    "First, let's do some research about solar panels. According to [1876 Energy](https://www.1876energy.com/5-factors-that-affect-solar-panel-efficiency/) and [Trace Software](https://www.trace-software.com/blog/which-are-the-factors-that-affect-solar-panels-efficiency/), the highest contributing factors to solar panels are temperature, energy conversion efficiency (power), shade, solar radiation, and location (longitude and latitude). Additionally, solar panels work more efficiently in cold temperatures, allowing the panel to produce more voltage and more electricity. Rain and snow have no effect on solar panels however cloudy days and humidity can slow down production."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step I: EDA\n",
    "First, we will perform some EDA so that we can get a feel for the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_set = pd.read_csv(\"cahsi_data_2020/D1.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>weather_datetime</th>\n",
       "      <th>solar_datetime</th>\n",
       "      <th>solarRadiation</th>\n",
       "      <th>uvHigh</th>\n",
       "      <th>winddirAvg</th>\n",
       "      <th>humidityHigh</th>\n",
       "      <th>humidityLow</th>\n",
       "      <th>humidityAvg</th>\n",
       "      <th>qcStatus</th>\n",
       "      <th>tempHigh</th>\n",
       "      <th>...</th>\n",
       "      <th>windchillAvg</th>\n",
       "      <th>heatindexHigh</th>\n",
       "      <th>heatindexLow</th>\n",
       "      <th>heatindexAvg</th>\n",
       "      <th>pressureMax</th>\n",
       "      <th>pressureMin</th>\n",
       "      <th>pressureTrend</th>\n",
       "      <th>precipRate</th>\n",
       "      <th>precipTotal</th>\n",
       "      <th>DC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2020-02-07 14:29:00</td>\n",
       "      <td>2020-02-07 14:29:1</td>\n",
       "      <td>627.70</td>\n",
       "      <td>7.0</td>\n",
       "      <td>195</td>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>-1</td>\n",
       "      <td>65</td>\n",
       "      <td>...</td>\n",
       "      <td>65</td>\n",
       "      <td>65</td>\n",
       "      <td>65</td>\n",
       "      <td>65</td>\n",
       "      <td>30.06</td>\n",
       "      <td>30.05</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42.036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2020-02-07 14:34:00</td>\n",
       "      <td>2020-02-07 14:34:1</td>\n",
       "      <td>617.31</td>\n",
       "      <td>7.0</td>\n",
       "      <td>129</td>\n",
       "      <td>24</td>\n",
       "      <td>23</td>\n",
       "      <td>23</td>\n",
       "      <td>-1</td>\n",
       "      <td>68</td>\n",
       "      <td>...</td>\n",
       "      <td>67</td>\n",
       "      <td>68</td>\n",
       "      <td>66</td>\n",
       "      <td>67</td>\n",
       "      <td>30.06</td>\n",
       "      <td>30.05</td>\n",
       "      <td>-0.15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42.126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2020-02-07 14:39:00</td>\n",
       "      <td>2020-02-07 14:39:1</td>\n",
       "      <td>608.13</td>\n",
       "      <td>6.0</td>\n",
       "      <td>108</td>\n",
       "      <td>24</td>\n",
       "      <td>23</td>\n",
       "      <td>23</td>\n",
       "      <td>-1</td>\n",
       "      <td>68</td>\n",
       "      <td>...</td>\n",
       "      <td>67</td>\n",
       "      <td>68</td>\n",
       "      <td>67</td>\n",
       "      <td>67</td>\n",
       "      <td>30.06</td>\n",
       "      <td>30.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42.264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2020-02-07 14:44:00</td>\n",
       "      <td>2020-02-07 14:44:1</td>\n",
       "      <td>582.57</td>\n",
       "      <td>6.0</td>\n",
       "      <td>87</td>\n",
       "      <td>25</td>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>-1</td>\n",
       "      <td>67</td>\n",
       "      <td>...</td>\n",
       "      <td>66</td>\n",
       "      <td>67</td>\n",
       "      <td>66</td>\n",
       "      <td>66</td>\n",
       "      <td>30.06</td>\n",
       "      <td>30.05</td>\n",
       "      <td>-0.15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42.204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2020-02-07 14:49:00</td>\n",
       "      <td>2020-02-07 14:49:1</td>\n",
       "      <td>571.67</td>\n",
       "      <td>6.0</td>\n",
       "      <td>38</td>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>-1</td>\n",
       "      <td>66</td>\n",
       "      <td>...</td>\n",
       "      <td>66</td>\n",
       "      <td>66</td>\n",
       "      <td>66</td>\n",
       "      <td>66</td>\n",
       "      <td>30.05</td>\n",
       "      <td>30.04</td>\n",
       "      <td>-0.15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>42.360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>95</td>\n",
       "      <td>2020-02-07 22:24:00</td>\n",
       "      <td>2020-02-07 22:24:1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>41</td>\n",
       "      <td>40</td>\n",
       "      <td>40</td>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>...</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>30.15</td>\n",
       "      <td>30.14</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>96</td>\n",
       "      <td>2020-02-07 22:29:00</td>\n",
       "      <td>2020-02-07 22:29:1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>43</td>\n",
       "      <td>41</td>\n",
       "      <td>42</td>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>...</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>50</td>\n",
       "      <td>51</td>\n",
       "      <td>30.15</td>\n",
       "      <td>30.14</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>97</td>\n",
       "      <td>2020-02-07 22:34:00</td>\n",
       "      <td>2020-02-07 22:34:1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>299</td>\n",
       "      <td>42</td>\n",
       "      <td>40</td>\n",
       "      <td>41</td>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>...</td>\n",
       "      <td>50</td>\n",
       "      <td>51</td>\n",
       "      <td>50</td>\n",
       "      <td>50</td>\n",
       "      <td>30.15</td>\n",
       "      <td>30.14</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>98</td>\n",
       "      <td>2020-02-07 22:39:00</td>\n",
       "      <td>2020-02-07 22:39:1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>233</td>\n",
       "      <td>42</td>\n",
       "      <td>41</td>\n",
       "      <td>41</td>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>...</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>30.15</td>\n",
       "      <td>30.15</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>99</td>\n",
       "      <td>2020-02-07 22:44:00</td>\n",
       "      <td>2020-02-07 22:44:1</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>248</td>\n",
       "      <td>41</td>\n",
       "      <td>39</td>\n",
       "      <td>40</td>\n",
       "      <td>1</td>\n",
       "      <td>51</td>\n",
       "      <td>...</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>51</td>\n",
       "      <td>30.16</td>\n",
       "      <td>30.15</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.198</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       weather_datetime      solar_datetime  solarRadiation  uvHigh  \\\n",
       "0   2020-02-07 14:29:00  2020-02-07 14:29:1          627.70     7.0   \n",
       "1   2020-02-07 14:34:00  2020-02-07 14:34:1          617.31     7.0   \n",
       "2   2020-02-07 14:39:00  2020-02-07 14:39:1          608.13     6.0   \n",
       "3   2020-02-07 14:44:00  2020-02-07 14:44:1          582.57     6.0   \n",
       "4   2020-02-07 14:49:00  2020-02-07 14:49:1          571.67     6.0   \n",
       "..                  ...                 ...             ...     ...   \n",
       "95  2020-02-07 22:24:00  2020-02-07 22:24:1            0.00     0.0   \n",
       "96  2020-02-07 22:29:00  2020-02-07 22:29:1            0.00     0.0   \n",
       "97  2020-02-07 22:34:00  2020-02-07 22:34:1            0.00     0.0   \n",
       "98  2020-02-07 22:39:00  2020-02-07 22:39:1            0.00     0.0   \n",
       "99  2020-02-07 22:44:00  2020-02-07 22:44:1            0.00     0.0   \n",
       "\n",
       "    winddirAvg  humidityHigh  humidityLow  humidityAvg  qcStatus  tempHigh  \\\n",
       "0          195            24           24           24        -1        65   \n",
       "1          129            24           23           23        -1        68   \n",
       "2          108            24           23           23        -1        68   \n",
       "3           87            25           24           24        -1        67   \n",
       "4           38            24           24           24        -1        66   \n",
       "..         ...           ...          ...          ...       ...       ...   \n",
       "95         255            41           40           40         1        51   \n",
       "96           3            43           41           42         1        51   \n",
       "97         299            42           40           41         1        51   \n",
       "98         233            42           41           41         1        51   \n",
       "99         248            41           39           40         1        51   \n",
       "\n",
       "    ...  windchillAvg  heatindexHigh  heatindexLow  heatindexAvg  pressureMax  \\\n",
       "0   ...            65             65            65            65        30.06   \n",
       "1   ...            67             68            66            67        30.06   \n",
       "2   ...            67             68            67            67        30.06   \n",
       "3   ...            66             67            66            66        30.06   \n",
       "4   ...            66             66            66            66        30.05   \n",
       "..  ...           ...            ...           ...           ...          ...   \n",
       "95  ...            51             51            51            51        30.15   \n",
       "96  ...            51             51            50            51        30.15   \n",
       "97  ...            50             51            50            50        30.15   \n",
       "98  ...            51             51            51            51        30.15   \n",
       "99  ...            51             51            51            51        30.16   \n",
       "\n",
       "    pressureMin  pressureTrend  precipRate  precipTotal      DC  \n",
       "0         30.05           0.60         0.0          0.0  42.036  \n",
       "1         30.05          -0.15         0.0          0.0  42.126  \n",
       "2         30.05           0.00         0.0          0.0  42.264  \n",
       "3         30.05          -0.15         0.0          0.0  42.204  \n",
       "4         30.04          -0.15         0.0          0.0  42.360  \n",
       "..          ...            ...         ...          ...     ...  \n",
       "95        30.14           0.15         0.0          0.0   0.186  \n",
       "96        30.14           0.15         0.0          0.0   0.192  \n",
       "97        30.14           0.00         0.0          0.0   0.192  \n",
       "98        30.15           0.00         0.0          0.0   0.192  \n",
       "99        30.15           0.00         0.0          0.0   0.198  \n",
       "\n",
       "[100 rows x 29 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_set.head(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>weather_datetime</th>\n",
       "      <th>solar_datetime</th>\n",
       "      <th>solarRadiation</th>\n",
       "      <th>uvHigh</th>\n",
       "      <th>winddirAvg</th>\n",
       "      <th>humidityHigh</th>\n",
       "      <th>humidityLow</th>\n",
       "      <th>humidityAvg</th>\n",
       "      <th>qcStatus</th>\n",
       "      <th>tempHigh</th>\n",
       "      <th>...</th>\n",
       "      <th>windchillAvg</th>\n",
       "      <th>heatindexHigh</th>\n",
       "      <th>heatindexLow</th>\n",
       "      <th>heatindexAvg</th>\n",
       "      <th>pressureMax</th>\n",
       "      <th>pressureMin</th>\n",
       "      <th>pressureTrend</th>\n",
       "      <th>precipRate</th>\n",
       "      <th>precipTotal</th>\n",
       "      <th>DC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>7955</td>\n",
       "      <td>2020-03-30 21:29:00</td>\n",
       "      <td>2020-03-30 21:29:1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>153</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>62</td>\n",
       "      <td>...</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>30.25</td>\n",
       "      <td>30.24</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7956</td>\n",
       "      <td>2020-03-30 21:34:00</td>\n",
       "      <td>2020-03-30 21:34:1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>160</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>62</td>\n",
       "      <td>...</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>30.25</td>\n",
       "      <td>30.24</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7957</td>\n",
       "      <td>2020-03-30 21:39:00</td>\n",
       "      <td>2020-03-30 21:39:1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>188</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>62</td>\n",
       "      <td>...</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>30.25</td>\n",
       "      <td>30.24</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7958</td>\n",
       "      <td>2020-03-30 21:44:00</td>\n",
       "      <td>2020-03-30 21:44:1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>153</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>62</td>\n",
       "      <td>...</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>30.25</td>\n",
       "      <td>30.24</td>\n",
       "      <td>-0.15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7959</td>\n",
       "      <td>2020-03-30 21:49:00</td>\n",
       "      <td>2020-03-30 21:49:1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>107</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>62</td>\n",
       "      <td>...</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>62</td>\n",
       "      <td>30.25</td>\n",
       "      <td>30.25</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.024</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         weather_datetime      solar_datetime  solarRadiation  uvHigh  \\\n",
       "7955  2020-03-30 21:29:00  2020-03-30 21:29:1             0.0     0.0   \n",
       "7956  2020-03-30 21:34:00  2020-03-30 21:34:1             0.0     0.0   \n",
       "7957  2020-03-30 21:39:00  2020-03-30 21:39:1             0.0     0.0   \n",
       "7958  2020-03-30 21:44:00  2020-03-30 21:44:1             0.0     0.0   \n",
       "7959  2020-03-30 21:49:00  2020-03-30 21:49:1             0.0     0.0   \n",
       "\n",
       "      winddirAvg  humidityHigh  humidityLow  humidityAvg  qcStatus  tempHigh  \\\n",
       "7955         153            25           25           25         1        62   \n",
       "7956         160            25           25           25         1        62   \n",
       "7957         188            25           25           25         1        62   \n",
       "7958         153            25           25           25         1        62   \n",
       "7959         107            25           25           25         1        62   \n",
       "\n",
       "      ...  windchillAvg  heatindexHigh  heatindexLow  heatindexAvg  \\\n",
       "7955  ...            62             62            62            62   \n",
       "7956  ...            62             62            62            62   \n",
       "7957  ...            62             62            62            62   \n",
       "7958  ...            62             62            62            62   \n",
       "7959  ...            62             62            62            62   \n",
       "\n",
       "      pressureMax  pressureMin  pressureTrend  precipRate  precipTotal     DC  \n",
       "7955        30.25        30.24           0.00         0.0          0.0  0.030  \n",
       "7956        30.25        30.24           0.00         0.0          0.0  0.024  \n",
       "7957        30.25        30.24           0.00         0.0          0.0  0.030  \n",
       "7958        30.25        30.24          -0.15         0.0          0.0  0.024  \n",
       "7959        30.25        30.25           0.00         0.0          0.0  0.024  \n",
       "\n",
       "[5 rows x 29 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_set.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observation: Notice that as it becomes later in the day, the solar radiation, uv, and temperature decreases. The DC voltage also decreases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['weather_datetime', 'solar_datetime', 'solarRadiation', 'uvHigh',\n",
       "       'winddirAvg', 'humidityHigh', 'humidityLow', 'humidityAvg', 'qcStatus',\n",
       "       'tempHigh', 'tempLow', 'tempAvg', 'windspeedHigh', 'windgustLow',\n",
       "       'windspeedAvg', 'dewptHigh', 'dewptLow', 'dewptAvg', 'windchillHigh',\n",
       "       'windchillAvg', 'heatindexHigh', 'heatindexLow', 'heatindexAvg',\n",
       "       'pressureMax', 'pressureMin', 'pressureTrend', 'precipRate',\n",
       "       'precipTotal', 'DC'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# what other columns are there?\n",
    "data_set.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7960, 29)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# what's the size of our data?\n",
    "data_set.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>solarRadiation</th>\n",
       "      <th>uvHigh</th>\n",
       "      <th>winddirAvg</th>\n",
       "      <th>humidityHigh</th>\n",
       "      <th>humidityLow</th>\n",
       "      <th>humidityAvg</th>\n",
       "      <th>qcStatus</th>\n",
       "      <th>tempHigh</th>\n",
       "      <th>tempLow</th>\n",
       "      <th>tempAvg</th>\n",
       "      <th>...</th>\n",
       "      <th>windchillAvg</th>\n",
       "      <th>heatindexHigh</th>\n",
       "      <th>heatindexLow</th>\n",
       "      <th>heatindexAvg</th>\n",
       "      <th>pressureMax</th>\n",
       "      <th>pressureMin</th>\n",
       "      <th>pressureTrend</th>\n",
       "      <th>precipRate</th>\n",
       "      <th>precipTotal</th>\n",
       "      <th>DC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>count</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "      <td>7960.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>mean</td>\n",
       "      <td>180.382851</td>\n",
       "      <td>1.844472</td>\n",
       "      <td>182.790075</td>\n",
       "      <td>45.861683</td>\n",
       "      <td>44.912437</td>\n",
       "      <td>45.077261</td>\n",
       "      <td>0.893970</td>\n",
       "      <td>53.745729</td>\n",
       "      <td>53.420603</td>\n",
       "      <td>53.554397</td>\n",
       "      <td>...</td>\n",
       "      <td>53.418970</td>\n",
       "      <td>53.581533</td>\n",
       "      <td>53.236935</td>\n",
       "      <td>53.380653</td>\n",
       "      <td>30.200309</td>\n",
       "      <td>30.193024</td>\n",
       "      <td>0.000974</td>\n",
       "      <td>0.000469</td>\n",
       "      <td>0.012936</td>\n",
       "      <td>19.539283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>std</td>\n",
       "      <td>264.082275</td>\n",
       "      <td>2.846040</td>\n",
       "      <td>78.432376</td>\n",
       "      <td>21.862087</td>\n",
       "      <td>21.940977</td>\n",
       "      <td>21.924786</td>\n",
       "      <td>0.311143</td>\n",
       "      <td>11.622671</td>\n",
       "      <td>11.565509</td>\n",
       "      <td>11.589908</td>\n",
       "      <td>...</td>\n",
       "      <td>11.720226</td>\n",
       "      <td>11.353402</td>\n",
       "      <td>11.265849</td>\n",
       "      <td>11.305647</td>\n",
       "      <td>0.137469</td>\n",
       "      <td>0.137532</td>\n",
       "      <td>0.095205</td>\n",
       "      <td>0.005050</td>\n",
       "      <td>0.053583</td>\n",
       "      <td>19.753129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>min</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>26.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>29.850000</td>\n",
       "      <td>29.830000</td>\n",
       "      <td>-0.600000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25%</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>139.000000</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>30.100000</td>\n",
       "      <td>30.100000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.030000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50%</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>197.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>42.000000</td>\n",
       "      <td>42.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>54.000000</td>\n",
       "      <td>54.000000</td>\n",
       "      <td>54.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>54.000000</td>\n",
       "      <td>54.000000</td>\n",
       "      <td>54.000000</td>\n",
       "      <td>54.000000</td>\n",
       "      <td>30.180000</td>\n",
       "      <td>30.180000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.379000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>75%</td>\n",
       "      <td>333.120000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>215.000000</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>30.250000</td>\n",
       "      <td>30.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>39.823500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>max</td>\n",
       "      <td>986.880000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>359.000000</td>\n",
       "      <td>98.000000</td>\n",
       "      <td>98.000000</td>\n",
       "      <td>98.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>83.000000</td>\n",
       "      <td>83.000000</td>\n",
       "      <td>83.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>83.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>30.610000</td>\n",
       "      <td>30.600000</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.130000</td>\n",
       "      <td>0.370000</td>\n",
       "      <td>43.710000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       solarRadiation       uvHigh   winddirAvg  humidityHigh  humidityLow  \\\n",
       "count     7960.000000  7960.000000  7960.000000   7960.000000  7960.000000   \n",
       "mean       180.382851     1.844472   182.790075     45.861683    44.912437   \n",
       "std        264.082275     2.846040    78.432376     21.862087    21.940977   \n",
       "min          0.000000     0.000000     0.000000     11.000000    10.000000   \n",
       "25%          0.000000     0.000000   139.000000     28.000000    27.000000   \n",
       "50%          0.000000     0.000000   197.000000     43.000000    42.000000   \n",
       "75%        333.120000     3.000000   215.000000     60.000000    59.000000   \n",
       "max        986.880000    10.000000   359.000000     98.000000    98.000000   \n",
       "\n",
       "       humidityAvg     qcStatus     tempHigh      tempLow      tempAvg  ...  \\\n",
       "count  7960.000000  7960.000000  7960.000000  7960.000000  7960.000000  ...   \n",
       "mean     45.077261     0.893970    53.745729    53.420603    53.554397  ...   \n",
       "std      21.924786     0.311143    11.622671    11.565509    11.589908  ...   \n",
       "min      10.000000    -1.000000    27.000000    27.000000    27.000000  ...   \n",
       "25%      27.000000     1.000000    45.000000    45.000000    45.000000  ...   \n",
       "50%      42.000000     1.000000    54.000000    54.000000    54.000000  ...   \n",
       "75%      60.000000     1.000000    62.000000    62.000000    62.000000  ...   \n",
       "max      98.000000     1.000000    83.000000    83.000000    83.000000  ...   \n",
       "\n",
       "       windchillAvg  heatindexHigh  heatindexLow  heatindexAvg  pressureMax  \\\n",
       "count   7960.000000    7960.000000   7960.000000   7960.000000  7960.000000   \n",
       "mean      53.418970      53.581533     53.236935     53.380653    30.200309   \n",
       "std       11.720226      11.353402     11.265849     11.305647     0.137469   \n",
       "min       26.000000      27.000000     27.000000     27.000000    29.850000   \n",
       "25%       45.000000      45.000000     45.000000     45.000000    30.100000   \n",
       "50%       54.000000      54.000000     54.000000     54.000000    30.180000   \n",
       "75%       62.000000      62.000000     62.000000     62.000000    30.250000   \n",
       "max       83.000000      80.000000     80.000000     80.000000    30.610000   \n",
       "\n",
       "       pressureMin  pressureTrend   precipRate  precipTotal           DC  \n",
       "count  7960.000000    7960.000000  7960.000000  7960.000000  7960.000000  \n",
       "mean     30.193024       0.000974     0.000469     0.012936    19.539283  \n",
       "std       0.137532       0.095205     0.005050     0.053583    19.753129  \n",
       "min      29.830000      -0.600000     0.000000     0.000000     0.000000  \n",
       "25%      30.100000       0.000000     0.000000     0.000000     0.030000  \n",
       "50%      30.180000       0.000000     0.000000     0.000000     8.379000  \n",
       "75%      30.250000       0.000000     0.000000     0.000000    39.823500  \n",
       "max      30.600000       0.600000     0.130000     0.370000    43.710000  \n",
       "\n",
       "[8 rows x 27 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how distributed is the data?\n",
    "data_set.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>solarRadiation</th>\n",
       "      <th>uvHigh</th>\n",
       "      <th>winddirAvg</th>\n",
       "      <th>humidityHigh</th>\n",
       "      <th>humidityLow</th>\n",
       "      <th>humidityAvg</th>\n",
       "      <th>qcStatus</th>\n",
       "      <th>tempHigh</th>\n",
       "      <th>tempLow</th>\n",
       "      <th>tempAvg</th>\n",
       "      <th>...</th>\n",
       "      <th>windchillAvg</th>\n",
       "      <th>heatindexHigh</th>\n",
       "      <th>heatindexLow</th>\n",
       "      <th>heatindexAvg</th>\n",
       "      <th>pressureMax</th>\n",
       "      <th>pressureMin</th>\n",
       "      <th>pressureTrend</th>\n",
       "      <th>precipRate</th>\n",
       "      <th>precipTotal</th>\n",
       "      <th>DC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>solarRadiation</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.937831</td>\n",
       "      <td>-0.342999</td>\n",
       "      <td>-0.399324</td>\n",
       "      <td>-0.408921</td>\n",
       "      <td>-0.405694</td>\n",
       "      <td>-0.051987</td>\n",
       "      <td>0.452500</td>\n",
       "      <td>0.444020</td>\n",
       "      <td>0.448101</td>\n",
       "      <td>...</td>\n",
       "      <td>0.446924</td>\n",
       "      <td>0.452235</td>\n",
       "      <td>0.442935</td>\n",
       "      <td>0.447495</td>\n",
       "      <td>0.046178</td>\n",
       "      <td>0.045571</td>\n",
       "      <td>-0.073512</td>\n",
       "      <td>0.001359</td>\n",
       "      <td>0.035785</td>\n",
       "      <td>0.814947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>uvHigh</td>\n",
       "      <td>0.937831</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.316591</td>\n",
       "      <td>-0.407405</td>\n",
       "      <td>-0.418103</td>\n",
       "      <td>-0.414418</td>\n",
       "      <td>-0.048438</td>\n",
       "      <td>0.454691</td>\n",
       "      <td>0.445485</td>\n",
       "      <td>0.449924</td>\n",
       "      <td>...</td>\n",
       "      <td>0.448278</td>\n",
       "      <td>0.454565</td>\n",
       "      <td>0.444371</td>\n",
       "      <td>0.449403</td>\n",
       "      <td>0.062698</td>\n",
       "      <td>0.062510</td>\n",
       "      <td>-0.096479</td>\n",
       "      <td>-0.043045</td>\n",
       "      <td>0.006510</td>\n",
       "      <td>0.700912</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>winddirAvg</td>\n",
       "      <td>-0.342999</td>\n",
       "      <td>-0.316591</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.319881</td>\n",
       "      <td>0.321128</td>\n",
       "      <td>0.320633</td>\n",
       "      <td>-0.058951</td>\n",
       "      <td>-0.362586</td>\n",
       "      <td>-0.361985</td>\n",
       "      <td>-0.362065</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.361569</td>\n",
       "      <td>-0.362126</td>\n",
       "      <td>-0.361636</td>\n",
       "      <td>-0.361757</td>\n",
       "      <td>0.196054</td>\n",
       "      <td>0.195846</td>\n",
       "      <td>0.011043</td>\n",
       "      <td>-0.018541</td>\n",
       "      <td>0.003610</td>\n",
       "      <td>-0.259023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>humidityHigh</td>\n",
       "      <td>-0.399324</td>\n",
       "      <td>-0.407405</td>\n",
       "      <td>0.319881</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.998860</td>\n",
       "      <td>0.999456</td>\n",
       "      <td>-0.132945</td>\n",
       "      <td>-0.765681</td>\n",
       "      <td>-0.763568</td>\n",
       "      <td>-0.764586</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.759300</td>\n",
       "      <td>-0.765219</td>\n",
       "      <td>-0.763032</td>\n",
       "      <td>-0.764093</td>\n",
       "      <td>0.169556</td>\n",
       "      <td>0.169504</td>\n",
       "      <td>0.027204</td>\n",
       "      <td>0.197191</td>\n",
       "      <td>0.362307</td>\n",
       "      <td>-0.102328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>humidityLow</td>\n",
       "      <td>-0.408921</td>\n",
       "      <td>-0.418103</td>\n",
       "      <td>0.321128</td>\n",
       "      <td>0.998860</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.999736</td>\n",
       "      <td>-0.132788</td>\n",
       "      <td>-0.767797</td>\n",
       "      <td>-0.765245</td>\n",
       "      <td>-0.766474</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.761256</td>\n",
       "      <td>-0.767374</td>\n",
       "      <td>-0.764697</td>\n",
       "      <td>-0.765993</td>\n",
       "      <td>0.167956</td>\n",
       "      <td>0.167909</td>\n",
       "      <td>0.029168</td>\n",
       "      <td>0.196765</td>\n",
       "      <td>0.360056</td>\n",
       "      <td>-0.109135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>humidityAvg</td>\n",
       "      <td>-0.405694</td>\n",
       "      <td>-0.414418</td>\n",
       "      <td>0.320633</td>\n",
       "      <td>0.999456</td>\n",
       "      <td>0.999736</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.132995</td>\n",
       "      <td>-0.766938</td>\n",
       "      <td>-0.764562</td>\n",
       "      <td>-0.765708</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.760461</td>\n",
       "      <td>-0.766498</td>\n",
       "      <td>-0.764012</td>\n",
       "      <td>-0.765218</td>\n",
       "      <td>0.168278</td>\n",
       "      <td>0.168241</td>\n",
       "      <td>0.028430</td>\n",
       "      <td>0.196955</td>\n",
       "      <td>0.361020</td>\n",
       "      <td>-0.106906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>qcStatus</td>\n",
       "      <td>-0.051987</td>\n",
       "      <td>-0.048438</td>\n",
       "      <td>-0.058951</td>\n",
       "      <td>-0.132945</td>\n",
       "      <td>-0.132788</td>\n",
       "      <td>-0.132995</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.050799</td>\n",
       "      <td>0.052421</td>\n",
       "      <td>0.051585</td>\n",
       "      <td>...</td>\n",
       "      <td>0.058814</td>\n",
       "      <td>0.050671</td>\n",
       "      <td>0.052195</td>\n",
       "      <td>0.051358</td>\n",
       "      <td>-0.130895</td>\n",
       "      <td>-0.131655</td>\n",
       "      <td>-0.013845</td>\n",
       "      <td>0.023923</td>\n",
       "      <td>0.051293</td>\n",
       "      <td>-0.128036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>tempHigh</td>\n",
       "      <td>0.452500</td>\n",
       "      <td>0.454691</td>\n",
       "      <td>-0.362586</td>\n",
       "      <td>-0.765681</td>\n",
       "      <td>-0.767797</td>\n",
       "      <td>-0.766938</td>\n",
       "      <td>0.050799</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.999030</td>\n",
       "      <td>0.999402</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998271</td>\n",
       "      <td>0.999708</td>\n",
       "      <td>0.998698</td>\n",
       "      <td>0.999141</td>\n",
       "      <td>-0.451769</td>\n",
       "      <td>-0.452192</td>\n",
       "      <td>-0.045211</td>\n",
       "      <td>-0.126698</td>\n",
       "      <td>-0.170193</td>\n",
       "      <td>0.176902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>tempLow</td>\n",
       "      <td>0.444020</td>\n",
       "      <td>0.445485</td>\n",
       "      <td>-0.361985</td>\n",
       "      <td>-0.763568</td>\n",
       "      <td>-0.765245</td>\n",
       "      <td>-0.764562</td>\n",
       "      <td>0.052421</td>\n",
       "      <td>0.999030</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.999544</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998397</td>\n",
       "      <td>0.998736</td>\n",
       "      <td>0.999677</td>\n",
       "      <td>0.999288</td>\n",
       "      <td>-0.455311</td>\n",
       "      <td>-0.455749</td>\n",
       "      <td>-0.043467</td>\n",
       "      <td>-0.125614</td>\n",
       "      <td>-0.170110</td>\n",
       "      <td>0.170358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>tempAvg</td>\n",
       "      <td>0.448101</td>\n",
       "      <td>0.449924</td>\n",
       "      <td>-0.362065</td>\n",
       "      <td>-0.764586</td>\n",
       "      <td>-0.766474</td>\n",
       "      <td>-0.765708</td>\n",
       "      <td>0.051585</td>\n",
       "      <td>0.999402</td>\n",
       "      <td>0.999544</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998819</td>\n",
       "      <td>0.999124</td>\n",
       "      <td>0.999221</td>\n",
       "      <td>0.999748</td>\n",
       "      <td>-0.453805</td>\n",
       "      <td>-0.454220</td>\n",
       "      <td>-0.044260</td>\n",
       "      <td>-0.126174</td>\n",
       "      <td>-0.170251</td>\n",
       "      <td>0.173594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>windspeedHigh</td>\n",
       "      <td>0.397328</td>\n",
       "      <td>0.383542</td>\n",
       "      <td>-0.321357</td>\n",
       "      <td>-0.520153</td>\n",
       "      <td>-0.516107</td>\n",
       "      <td>-0.517716</td>\n",
       "      <td>-0.011902</td>\n",
       "      <td>0.500599</td>\n",
       "      <td>0.503205</td>\n",
       "      <td>0.502059</td>\n",
       "      <td>...</td>\n",
       "      <td>0.484797</td>\n",
       "      <td>0.500218</td>\n",
       "      <td>0.502660</td>\n",
       "      <td>0.501699</td>\n",
       "      <td>-0.048984</td>\n",
       "      <td>-0.049268</td>\n",
       "      <td>-0.030898</td>\n",
       "      <td>-0.107910</td>\n",
       "      <td>-0.222856</td>\n",
       "      <td>0.242005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>windgustLow</td>\n",
       "      <td>0.330530</td>\n",
       "      <td>0.317268</td>\n",
       "      <td>-0.276300</td>\n",
       "      <td>-0.407748</td>\n",
       "      <td>-0.403040</td>\n",
       "      <td>-0.404775</td>\n",
       "      <td>-0.000308</td>\n",
       "      <td>0.397769</td>\n",
       "      <td>0.401008</td>\n",
       "      <td>0.399604</td>\n",
       "      <td>...</td>\n",
       "      <td>0.382369</td>\n",
       "      <td>0.397298</td>\n",
       "      <td>0.400612</td>\n",
       "      <td>0.399310</td>\n",
       "      <td>-0.041768</td>\n",
       "      <td>-0.041718</td>\n",
       "      <td>-0.019151</td>\n",
       "      <td>-0.086551</td>\n",
       "      <td>-0.173471</td>\n",
       "      <td>0.214411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>windspeedAvg</td>\n",
       "      <td>0.387350</td>\n",
       "      <td>0.372748</td>\n",
       "      <td>-0.316086</td>\n",
       "      <td>-0.489531</td>\n",
       "      <td>-0.485025</td>\n",
       "      <td>-0.486786</td>\n",
       "      <td>-0.003781</td>\n",
       "      <td>0.472159</td>\n",
       "      <td>0.475211</td>\n",
       "      <td>0.473931</td>\n",
       "      <td>...</td>\n",
       "      <td>0.456297</td>\n",
       "      <td>0.471763</td>\n",
       "      <td>0.474679</td>\n",
       "      <td>0.473568</td>\n",
       "      <td>-0.044306</td>\n",
       "      <td>-0.044478</td>\n",
       "      <td>-0.025686</td>\n",
       "      <td>-0.101311</td>\n",
       "      <td>-0.205448</td>\n",
       "      <td>0.245529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>dewptHigh</td>\n",
       "      <td>-0.052317</td>\n",
       "      <td>-0.050672</td>\n",
       "      <td>0.068162</td>\n",
       "      <td>0.567230</td>\n",
       "      <td>0.563345</td>\n",
       "      <td>0.565181</td>\n",
       "      <td>-0.143246</td>\n",
       "      <td>0.050101</td>\n",
       "      <td>0.052756</td>\n",
       "      <td>0.051704</td>\n",
       "      <td>...</td>\n",
       "      <td>0.057759</td>\n",
       "      <td>0.050857</td>\n",
       "      <td>0.053645</td>\n",
       "      <td>0.052601</td>\n",
       "      <td>-0.235636</td>\n",
       "      <td>-0.236398</td>\n",
       "      <td>-0.020742</td>\n",
       "      <td>0.119437</td>\n",
       "      <td>0.299852</td>\n",
       "      <td>0.049395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>dewptLow</td>\n",
       "      <td>-0.080215</td>\n",
       "      <td>-0.080862</td>\n",
       "      <td>0.079540</td>\n",
       "      <td>0.593152</td>\n",
       "      <td>0.592413</td>\n",
       "      <td>0.593069</td>\n",
       "      <td>-0.141533</td>\n",
       "      <td>0.017321</td>\n",
       "      <td>0.020574</td>\n",
       "      <td>0.019246</td>\n",
       "      <td>...</td>\n",
       "      <td>0.025345</td>\n",
       "      <td>0.018005</td>\n",
       "      <td>0.021514</td>\n",
       "      <td>0.020128</td>\n",
       "      <td>-0.227196</td>\n",
       "      <td>-0.227908</td>\n",
       "      <td>-0.017020</td>\n",
       "      <td>0.124784</td>\n",
       "      <td>0.308419</td>\n",
       "      <td>0.035530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>dewptAvg</td>\n",
       "      <td>-0.066127</td>\n",
       "      <td>-0.065882</td>\n",
       "      <td>0.074241</td>\n",
       "      <td>0.580213</td>\n",
       "      <td>0.577895</td>\n",
       "      <td>0.579206</td>\n",
       "      <td>-0.140929</td>\n",
       "      <td>0.034221</td>\n",
       "      <td>0.037206</td>\n",
       "      <td>0.036009</td>\n",
       "      <td>...</td>\n",
       "      <td>0.042133</td>\n",
       "      <td>0.034966</td>\n",
       "      <td>0.038134</td>\n",
       "      <td>0.036922</td>\n",
       "      <td>-0.231846</td>\n",
       "      <td>-0.232595</td>\n",
       "      <td>-0.018377</td>\n",
       "      <td>0.122124</td>\n",
       "      <td>0.304470</td>\n",
       "      <td>0.041853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>windchillHigh</td>\n",
       "      <td>0.452330</td>\n",
       "      <td>0.454327</td>\n",
       "      <td>-0.363098</td>\n",
       "      <td>-0.764098</td>\n",
       "      <td>-0.766250</td>\n",
       "      <td>-0.765379</td>\n",
       "      <td>0.055493</td>\n",
       "      <td>0.999509</td>\n",
       "      <td>0.998592</td>\n",
       "      <td>0.998945</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998978</td>\n",
       "      <td>0.999217</td>\n",
       "      <td>0.998260</td>\n",
       "      <td>0.998684</td>\n",
       "      <td>-0.453846</td>\n",
       "      <td>-0.454269</td>\n",
       "      <td>-0.045599</td>\n",
       "      <td>-0.126133</td>\n",
       "      <td>-0.168847</td>\n",
       "      <td>0.176803</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>windchillAvg</td>\n",
       "      <td>0.446924</td>\n",
       "      <td>0.448278</td>\n",
       "      <td>-0.361569</td>\n",
       "      <td>-0.759300</td>\n",
       "      <td>-0.761256</td>\n",
       "      <td>-0.760461</td>\n",
       "      <td>0.058814</td>\n",
       "      <td>0.998271</td>\n",
       "      <td>0.998397</td>\n",
       "      <td>0.998819</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.997993</td>\n",
       "      <td>0.998074</td>\n",
       "      <td>0.998567</td>\n",
       "      <td>-0.459162</td>\n",
       "      <td>-0.459578</td>\n",
       "      <td>-0.044587</td>\n",
       "      <td>-0.124342</td>\n",
       "      <td>-0.166472</td>\n",
       "      <td>0.173580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>heatindexHigh</td>\n",
       "      <td>0.452235</td>\n",
       "      <td>0.454565</td>\n",
       "      <td>-0.362126</td>\n",
       "      <td>-0.765219</td>\n",
       "      <td>-0.767374</td>\n",
       "      <td>-0.766498</td>\n",
       "      <td>0.050671</td>\n",
       "      <td>0.999708</td>\n",
       "      <td>0.998736</td>\n",
       "      <td>0.999124</td>\n",
       "      <td>...</td>\n",
       "      <td>0.997993</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.998743</td>\n",
       "      <td>0.999269</td>\n",
       "      <td>-0.452040</td>\n",
       "      <td>-0.452460</td>\n",
       "      <td>-0.045153</td>\n",
       "      <td>-0.126702</td>\n",
       "      <td>-0.170172</td>\n",
       "      <td>0.176960</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>heatindexLow</td>\n",
       "      <td>0.442935</td>\n",
       "      <td>0.444371</td>\n",
       "      <td>-0.361636</td>\n",
       "      <td>-0.763032</td>\n",
       "      <td>-0.764697</td>\n",
       "      <td>-0.764012</td>\n",
       "      <td>0.052195</td>\n",
       "      <td>0.998698</td>\n",
       "      <td>0.999677</td>\n",
       "      <td>0.999221</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998074</td>\n",
       "      <td>0.998743</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.999439</td>\n",
       "      <td>-0.455523</td>\n",
       "      <td>-0.455950</td>\n",
       "      <td>-0.043394</td>\n",
       "      <td>-0.125619</td>\n",
       "      <td>-0.170043</td>\n",
       "      <td>0.170265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>heatindexAvg</td>\n",
       "      <td>0.447495</td>\n",
       "      <td>0.449403</td>\n",
       "      <td>-0.361757</td>\n",
       "      <td>-0.764093</td>\n",
       "      <td>-0.765993</td>\n",
       "      <td>-0.765218</td>\n",
       "      <td>0.051358</td>\n",
       "      <td>0.999141</td>\n",
       "      <td>0.999288</td>\n",
       "      <td>0.999748</td>\n",
       "      <td>...</td>\n",
       "      <td>0.998567</td>\n",
       "      <td>0.999269</td>\n",
       "      <td>0.999439</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.454029</td>\n",
       "      <td>-0.454441</td>\n",
       "      <td>-0.044092</td>\n",
       "      <td>-0.126179</td>\n",
       "      <td>-0.170233</td>\n",
       "      <td>0.173561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>pressureMax</td>\n",
       "      <td>0.046178</td>\n",
       "      <td>0.062698</td>\n",
       "      <td>0.196054</td>\n",
       "      <td>0.169556</td>\n",
       "      <td>0.167956</td>\n",
       "      <td>0.168278</td>\n",
       "      <td>-0.130895</td>\n",
       "      <td>-0.451769</td>\n",
       "      <td>-0.455311</td>\n",
       "      <td>-0.453805</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.459162</td>\n",
       "      <td>-0.452040</td>\n",
       "      <td>-0.455523</td>\n",
       "      <td>-0.454029</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.998638</td>\n",
       "      <td>-0.016431</td>\n",
       "      <td>-0.101865</td>\n",
       "      <td>-0.224125</td>\n",
       "      <td>0.081685</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>pressureMin</td>\n",
       "      <td>0.045571</td>\n",
       "      <td>0.062510</td>\n",
       "      <td>0.195846</td>\n",
       "      <td>0.169504</td>\n",
       "      <td>0.167909</td>\n",
       "      <td>0.168241</td>\n",
       "      <td>-0.131655</td>\n",
       "      <td>-0.452192</td>\n",
       "      <td>-0.455749</td>\n",
       "      <td>-0.454220</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.459578</td>\n",
       "      <td>-0.452460</td>\n",
       "      <td>-0.455950</td>\n",
       "      <td>-0.454441</td>\n",
       "      <td>0.998638</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.016344</td>\n",
       "      <td>-0.103346</td>\n",
       "      <td>-0.224628</td>\n",
       "      <td>0.081169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>pressureTrend</td>\n",
       "      <td>-0.073512</td>\n",
       "      <td>-0.096479</td>\n",
       "      <td>0.011043</td>\n",
       "      <td>0.027204</td>\n",
       "      <td>0.029168</td>\n",
       "      <td>0.028430</td>\n",
       "      <td>-0.013845</td>\n",
       "      <td>-0.045211</td>\n",
       "      <td>-0.043467</td>\n",
       "      <td>-0.044260</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.044587</td>\n",
       "      <td>-0.045153</td>\n",
       "      <td>-0.043394</td>\n",
       "      <td>-0.044092</td>\n",
       "      <td>-0.016431</td>\n",
       "      <td>-0.016344</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.018658</td>\n",
       "      <td>0.018448</td>\n",
       "      <td>-0.027936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>precipRate</td>\n",
       "      <td>0.001359</td>\n",
       "      <td>-0.043045</td>\n",
       "      <td>-0.018541</td>\n",
       "      <td>0.197191</td>\n",
       "      <td>0.196765</td>\n",
       "      <td>0.196955</td>\n",
       "      <td>0.023923</td>\n",
       "      <td>-0.126698</td>\n",
       "      <td>-0.125614</td>\n",
       "      <td>-0.126174</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.124342</td>\n",
       "      <td>-0.126702</td>\n",
       "      <td>-0.125619</td>\n",
       "      <td>-0.126179</td>\n",
       "      <td>-0.101865</td>\n",
       "      <td>-0.103346</td>\n",
       "      <td>0.018658</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.410878</td>\n",
       "      <td>0.094109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>precipTotal</td>\n",
       "      <td>0.035785</td>\n",
       "      <td>0.006510</td>\n",
       "      <td>0.003610</td>\n",
       "      <td>0.362307</td>\n",
       "      <td>0.360056</td>\n",
       "      <td>0.361020</td>\n",
       "      <td>0.051293</td>\n",
       "      <td>-0.170193</td>\n",
       "      <td>-0.170110</td>\n",
       "      <td>-0.170251</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.166472</td>\n",
       "      <td>-0.170172</td>\n",
       "      <td>-0.170043</td>\n",
       "      <td>-0.170233</td>\n",
       "      <td>-0.224125</td>\n",
       "      <td>-0.224628</td>\n",
       "      <td>0.018448</td>\n",
       "      <td>0.410878</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.140794</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>DC</td>\n",
       "      <td>0.814947</td>\n",
       "      <td>0.700912</td>\n",
       "      <td>-0.259023</td>\n",
       "      <td>-0.102328</td>\n",
       "      <td>-0.109135</td>\n",
       "      <td>-0.106906</td>\n",
       "      <td>-0.128036</td>\n",
       "      <td>0.176902</td>\n",
       "      <td>0.170358</td>\n",
       "      <td>0.173594</td>\n",
       "      <td>...</td>\n",
       "      <td>0.173580</td>\n",
       "      <td>0.176960</td>\n",
       "      <td>0.170265</td>\n",
       "      <td>0.173561</td>\n",
       "      <td>0.081685</td>\n",
       "      <td>0.081169</td>\n",
       "      <td>-0.027936</td>\n",
       "      <td>0.094109</td>\n",
       "      <td>0.140794</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>27 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                solarRadiation    uvHigh  winddirAvg  humidityHigh  \\\n",
       "solarRadiation        1.000000  0.937831   -0.342999     -0.399324   \n",
       "uvHigh                0.937831  1.000000   -0.316591     -0.407405   \n",
       "winddirAvg           -0.342999 -0.316591    1.000000      0.319881   \n",
       "humidityHigh         -0.399324 -0.407405    0.319881      1.000000   \n",
       "humidityLow          -0.408921 -0.418103    0.321128      0.998860   \n",
       "humidityAvg          -0.405694 -0.414418    0.320633      0.999456   \n",
       "qcStatus             -0.051987 -0.048438   -0.058951     -0.132945   \n",
       "tempHigh              0.452500  0.454691   -0.362586     -0.765681   \n",
       "tempLow               0.444020  0.445485   -0.361985     -0.763568   \n",
       "tempAvg               0.448101  0.449924   -0.362065     -0.764586   \n",
       "windspeedHigh         0.397328  0.383542   -0.321357     -0.520153   \n",
       "windgustLow           0.330530  0.317268   -0.276300     -0.407748   \n",
       "windspeedAvg          0.387350  0.372748   -0.316086     -0.489531   \n",
       "dewptHigh            -0.052317 -0.050672    0.068162      0.567230   \n",
       "dewptLow             -0.080215 -0.080862    0.079540      0.593152   \n",
       "dewptAvg             -0.066127 -0.065882    0.074241      0.580213   \n",
       "windchillHigh         0.452330  0.454327   -0.363098     -0.764098   \n",
       "windchillAvg          0.446924  0.448278   -0.361569     -0.759300   \n",
       "heatindexHigh         0.452235  0.454565   -0.362126     -0.765219   \n",
       "heatindexLow          0.442935  0.444371   -0.361636     -0.763032   \n",
       "heatindexAvg          0.447495  0.449403   -0.361757     -0.764093   \n",
       "pressureMax           0.046178  0.062698    0.196054      0.169556   \n",
       "pressureMin           0.045571  0.062510    0.195846      0.169504   \n",
       "pressureTrend        -0.073512 -0.096479    0.011043      0.027204   \n",
       "precipRate            0.001359 -0.043045   -0.018541      0.197191   \n",
       "precipTotal           0.035785  0.006510    0.003610      0.362307   \n",
       "DC                    0.814947  0.700912   -0.259023     -0.102328   \n",
       "\n",
       "                humidityLow  humidityAvg  qcStatus  tempHigh   tempLow  \\\n",
       "solarRadiation    -0.408921    -0.405694 -0.051987  0.452500  0.444020   \n",
       "uvHigh            -0.418103    -0.414418 -0.048438  0.454691  0.445485   \n",
       "winddirAvg         0.321128     0.320633 -0.058951 -0.362586 -0.361985   \n",
       "humidityHigh       0.998860     0.999456 -0.132945 -0.765681 -0.763568   \n",
       "humidityLow        1.000000     0.999736 -0.132788 -0.767797 -0.765245   \n",
       "humidityAvg        0.999736     1.000000 -0.132995 -0.766938 -0.764562   \n",
       "qcStatus          -0.132788    -0.132995  1.000000  0.050799  0.052421   \n",
       "tempHigh          -0.767797    -0.766938  0.050799  1.000000  0.999030   \n",
       "tempLow           -0.765245    -0.764562  0.052421  0.999030  1.000000   \n",
       "tempAvg           -0.766474    -0.765708  0.051585  0.999402  0.999544   \n",
       "windspeedHigh     -0.516107    -0.517716 -0.011902  0.500599  0.503205   \n",
       "windgustLow       -0.403040    -0.404775 -0.000308  0.397769  0.401008   \n",
       "windspeedAvg      -0.485025    -0.486786 -0.003781  0.472159  0.475211   \n",
       "dewptHigh          0.563345     0.565181 -0.143246  0.050101  0.052756   \n",
       "dewptLow           0.592413     0.593069 -0.141533  0.017321  0.020574   \n",
       "dewptAvg           0.577895     0.579206 -0.140929  0.034221  0.037206   \n",
       "windchillHigh     -0.766250    -0.765379  0.055493  0.999509  0.998592   \n",
       "windchillAvg      -0.761256    -0.760461  0.058814  0.998271  0.998397   \n",
       "heatindexHigh     -0.767374    -0.766498  0.050671  0.999708  0.998736   \n",
       "heatindexLow      -0.764697    -0.764012  0.052195  0.998698  0.999677   \n",
       "heatindexAvg      -0.765993    -0.765218  0.051358  0.999141  0.999288   \n",
       "pressureMax        0.167956     0.168278 -0.130895 -0.451769 -0.455311   \n",
       "pressureMin        0.167909     0.168241 -0.131655 -0.452192 -0.455749   \n",
       "pressureTrend      0.029168     0.028430 -0.013845 -0.045211 -0.043467   \n",
       "precipRate         0.196765     0.196955  0.023923 -0.126698 -0.125614   \n",
       "precipTotal        0.360056     0.361020  0.051293 -0.170193 -0.170110   \n",
       "DC                -0.109135    -0.106906 -0.128036  0.176902  0.170358   \n",
       "\n",
       "                 tempAvg  ...  windchillAvg  heatindexHigh  heatindexLow  \\\n",
       "solarRadiation  0.448101  ...      0.446924       0.452235      0.442935   \n",
       "uvHigh          0.449924  ...      0.448278       0.454565      0.444371   \n",
       "winddirAvg     -0.362065  ...     -0.361569      -0.362126     -0.361636   \n",
       "humidityHigh   -0.764586  ...     -0.759300      -0.765219     -0.763032   \n",
       "humidityLow    -0.766474  ...     -0.761256      -0.767374     -0.764697   \n",
       "humidityAvg    -0.765708  ...     -0.760461      -0.766498     -0.764012   \n",
       "qcStatus        0.051585  ...      0.058814       0.050671      0.052195   \n",
       "tempHigh        0.999402  ...      0.998271       0.999708      0.998698   \n",
       "tempLow         0.999544  ...      0.998397       0.998736      0.999677   \n",
       "tempAvg         1.000000  ...      0.998819       0.999124      0.999221   \n",
       "windspeedHigh   0.502059  ...      0.484797       0.500218      0.502660   \n",
       "windgustLow     0.399604  ...      0.382369       0.397298      0.400612   \n",
       "windspeedAvg    0.473931  ...      0.456297       0.471763      0.474679   \n",
       "dewptHigh       0.051704  ...      0.057759       0.050857      0.053645   \n",
       "dewptLow        0.019246  ...      0.025345       0.018005      0.021514   \n",
       "dewptAvg        0.036009  ...      0.042133       0.034966      0.038134   \n",
       "windchillHigh   0.998945  ...      0.998978       0.999217      0.998260   \n",
       "windchillAvg    0.998819  ...      1.000000       0.997993      0.998074   \n",
       "heatindexHigh   0.999124  ...      0.997993       1.000000      0.998743   \n",
       "heatindexLow    0.999221  ...      0.998074       0.998743      1.000000   \n",
       "heatindexAvg    0.999748  ...      0.998567       0.999269      0.999439   \n",
       "pressureMax    -0.453805  ...     -0.459162      -0.452040     -0.455523   \n",
       "pressureMin    -0.454220  ...     -0.459578      -0.452460     -0.455950   \n",
       "pressureTrend  -0.044260  ...     -0.044587      -0.045153     -0.043394   \n",
       "precipRate     -0.126174  ...     -0.124342      -0.126702     -0.125619   \n",
       "precipTotal    -0.170251  ...     -0.166472      -0.170172     -0.170043   \n",
       "DC              0.173594  ...      0.173580       0.176960      0.170265   \n",
       "\n",
       "                heatindexAvg  pressureMax  pressureMin  pressureTrend  \\\n",
       "solarRadiation      0.447495     0.046178     0.045571      -0.073512   \n",
       "uvHigh              0.449403     0.062698     0.062510      -0.096479   \n",
       "winddirAvg         -0.361757     0.196054     0.195846       0.011043   \n",
       "humidityHigh       -0.764093     0.169556     0.169504       0.027204   \n",
       "humidityLow        -0.765993     0.167956     0.167909       0.029168   \n",
       "humidityAvg        -0.765218     0.168278     0.168241       0.028430   \n",
       "qcStatus            0.051358    -0.130895    -0.131655      -0.013845   \n",
       "tempHigh            0.999141    -0.451769    -0.452192      -0.045211   \n",
       "tempLow             0.999288    -0.455311    -0.455749      -0.043467   \n",
       "tempAvg             0.999748    -0.453805    -0.454220      -0.044260   \n",
       "windspeedHigh       0.501699    -0.048984    -0.049268      -0.030898   \n",
       "windgustLow         0.399310    -0.041768    -0.041718      -0.019151   \n",
       "windspeedAvg        0.473568    -0.044306    -0.044478      -0.025686   \n",
       "dewptHigh           0.052601    -0.235636    -0.236398      -0.020742   \n",
       "dewptLow            0.020128    -0.227196    -0.227908      -0.017020   \n",
       "dewptAvg            0.036922    -0.231846    -0.232595      -0.018377   \n",
       "windchillHigh       0.998684    -0.453846    -0.454269      -0.045599   \n",
       "windchillAvg        0.998567    -0.459162    -0.459578      -0.044587   \n",
       "heatindexHigh       0.999269    -0.452040    -0.452460      -0.045153   \n",
       "heatindexLow        0.999439    -0.455523    -0.455950      -0.043394   \n",
       "heatindexAvg        1.000000    -0.454029    -0.454441      -0.044092   \n",
       "pressureMax        -0.454029     1.000000     0.998638      -0.016431   \n",
       "pressureMin        -0.454441     0.998638     1.000000      -0.016344   \n",
       "pressureTrend      -0.044092    -0.016431    -0.016344       1.000000   \n",
       "precipRate         -0.126179    -0.101865    -0.103346       0.018658   \n",
       "precipTotal        -0.170233    -0.224125    -0.224628       0.018448   \n",
       "DC                  0.173561     0.081685     0.081169      -0.027936   \n",
       "\n",
       "                precipRate  precipTotal        DC  \n",
       "solarRadiation    0.001359     0.035785  0.814947  \n",
       "uvHigh           -0.043045     0.006510  0.700912  \n",
       "winddirAvg       -0.018541     0.003610 -0.259023  \n",
       "humidityHigh      0.197191     0.362307 -0.102328  \n",
       "humidityLow       0.196765     0.360056 -0.109135  \n",
       "humidityAvg       0.196955     0.361020 -0.106906  \n",
       "qcStatus          0.023923     0.051293 -0.128036  \n",
       "tempHigh         -0.126698    -0.170193  0.176902  \n",
       "tempLow          -0.125614    -0.170110  0.170358  \n",
       "tempAvg          -0.126174    -0.170251  0.173594  \n",
       "windspeedHigh    -0.107910    -0.222856  0.242005  \n",
       "windgustLow      -0.086551    -0.173471  0.214411  \n",
       "windspeedAvg     -0.101311    -0.205448  0.245529  \n",
       "dewptHigh         0.119437     0.299852  0.049395  \n",
       "dewptLow          0.124784     0.308419  0.035530  \n",
       "dewptAvg          0.122124     0.304470  0.041853  \n",
       "windchillHigh    -0.126133    -0.168847  0.176803  \n",
       "windchillAvg     -0.124342    -0.166472  0.173580  \n",
       "heatindexHigh    -0.126702    -0.170172  0.176960  \n",
       "heatindexLow     -0.125619    -0.170043  0.170265  \n",
       "heatindexAvg     -0.126179    -0.170233  0.173561  \n",
       "pressureMax      -0.101865    -0.224125  0.081685  \n",
       "pressureMin      -0.103346    -0.224628  0.081169  \n",
       "pressureTrend     0.018658     0.018448 -0.027936  \n",
       "precipRate        1.000000     0.410878  0.094109  \n",
       "precipTotal       0.410878     1.000000  0.140794  \n",
       "DC                0.094109     0.140794  1.000000  \n",
       "\n",
       "[27 rows x 27 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use pd.DataFrame.corr function to see what correlations can be identified between DC and other features.\n",
    "data_set.corr(method=\"spearman\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Observation__: In relation to DC, it appears there is a strong correlation with: \n",
    "* ```solarRadiation``` - 0.8\n",
    "* ```uvHigh``` - 0.7\n",
    "\n",
    "and loose correlation with:\n",
    "\n",
    "* ```tempHigh```\n",
    "* ```tempLow```\n",
    "* ```tempAvg```\n",
    "* ```windchillAvg```\n",
    "* ```heatindexHigh```\n",
    "* ```heatindexLow```\n",
    "* ```heatindexAvg```\n",
    "* ```precipTotal```\n",
    "\n",
    "Does this reflect any information gathered from our research?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step II: Feature Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will split the data into features and labels and convert them into arrays to be used for our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we want to perdict DC\n",
    "labels = np.array(data_set['DC'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the labels and unimportant features from the features list.\n",
    "\n",
    "col = [\n",
    " 'weather_datetime',\n",
    " 'solar_datetime',\n",
    " 'winddirAvg',\n",
    " 'humidityHigh',\n",
    " 'humidityLow',\n",
    " 'humidityAvg',\n",
    " 'heatindexLow',\n",
    " 'heatindexHigh',\n",
    " 'heatindexAvg',\n",
    " 'qcStatus',\n",
    " 'windspeedHigh',\n",
    " 'windgustLow',\n",
    " 'windspeedAvg',\n",
    " 'dewptHigh',\n",
    " 'dewptLow',\n",
    " 'dewptAvg',\n",
    " 'windchillHigh',\n",
    " 'windchillAvg',\n",
    " 'pressureMax',\n",
    " 'pressureMin',\n",
    " 'pressureTrend',\n",
    " 'precipRate',\n",
    " 'precipTotal',\n",
    " 'DC']\n",
    "\n",
    "features= data_set.drop(col, axis = 1)\n",
    "feature_list = list(features.columns)\n",
    "features = np.array(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step III: Build and Train Model\n",
    "Split the data into train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note here that the test size is so low because I want to overfit the model since we have a separate test set.\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size = 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Features Shape: (7164, 5)\n",
      "Training Labels Shape: (7164,)\n",
      "Testing Features Shape: (796, 5)\n",
      "Testing Labels Shape: (796,)\n"
     ]
    }
   ],
   "source": [
    "print('Training Features Shape:', train_features.shape)\n",
    "print('Training Labels Shape:', train_labels.shape)\n",
    "print('Testing Features Shape:', test_features.shape)\n",
    "print('Testing Labels Shape:', test_labels.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['solarRadiation', 'uvHigh', 'tempHigh', 'tempLow', 'tempAvg']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the features we will be using to predict DC\n",
    "feature_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step III.i: Hyper Parameters Tuning\n",
    "Hyper Parameters Tuning is good for figuring out what parameters will work the best for building the model. It's much better than guessing. Although it isn't perfect, it gives us some clues on what to try."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "\n",
    "from scipy.stats import uniform as sp_randFloat\n",
    "from scipy.stats import randint as sp_randInt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "```Gradient Boost```\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "gradient_boost_model = GradientBoostingRegressor()\n",
    "gradient_params = {'learning_rate': sp_randFloat(),\n",
    "                'subsample'    : sp_randFloat(),\n",
    "                'n_estimators' : sp_randInt(200, 2000),\n",
    "                'max_depth'    : sp_randInt(10, 110)\n",
    "             }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 100 candidates, totalling 300 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  33 tasks      | elapsed:  2.9min\n",
      "[Parallel(n_jobs=-1)]: Done 154 tasks      | elapsed: 14.9min\n",
      "[Parallel(n_jobs=-1)]: Done 300 out of 300 | elapsed: 28.2min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=3, estimator=GradientBoostingRegressor(), n_iter=100,\n",
       "                   n_jobs=-1,\n",
       "                   param_distributions={'learning_rate': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7fe6ae2aedd0>,\n",
       "                                        'max_depth': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7fe6ae2aecd0>,\n",
       "                                        'n_estimators': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7fe6ae2ae050>,\n",
       "                                        'subsample': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7fe6ae2aee10>},\n",
       "                   verbose=2)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_gradient = RandomizedSearchCV(estimator= gradient_boost_model, param_distributions = gradient_params, cv = 3, verbose=2, n_iter = 100, n_jobs=-1)\n",
    "random_gradient.fit(train_features, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Results from Random Search \n",
      "\n",
      " The best estimator across ALL searched params:\n",
      " GradientBoostingRegressor(learning_rate=0.01794706377831745, max_depth=32,\n",
      "                          n_estimators=785, subsample=0.2873167459093807)\n",
      "\n",
      " The best score across ALL searched params:\n",
      " 0.9655473309872132\n",
      "\n",
      " The best parameters across ALL searched params:\n",
      " {'learning_rate': 0.01794706377831745, 'max_depth': 32, 'n_estimators': 785, 'subsample': 0.2873167459093807}\n",
      "0.9688789434674687\n"
     ]
    }
   ],
   "source": [
    "# Results from Random Search\n",
    "print(\" Results from Random Search \" )\n",
    "print(\"\\n The best estimator across ALL searched params:\\n\", random_gradient.best_estimator_)\n",
    "print(\"\\n The best score across ALL searched params:\\n\", random_gradient.best_score_)\n",
    "print(\"\\n The best parameters across ALL searched params:\\n\", random_gradient.best_params_)\n",
    "print(random_gradient.score(test_features , test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step III.ii: Random Forest Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate model with 1500 decision trees\n",
    "rf = RandomForestRegressor(n_estimators = 785, \n",
    "                           criterion=\"mse\", \n",
    "                           max_depth = 32, \n",
    "                           min_samples_split = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestRegressor(max_depth=32, n_estimators=785)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train the model on training data\n",
    "rf.fit(train_features, train_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step III.iii: Accuracy - R2 Score\n",
    "Let's see what the accuracy our model is using the training set provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = rf.predict(test_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9709460517127418"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import r2_score\n",
    "\n",
    "r2_score(test_labels, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Comment__: Our model has a accuracy of 97%! That's not bad at all."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sept IV: Predictions Using Test Dataset\n",
    "Now we will test our model using the test set. Remember that whatever we did to the training set must also be done to the testing set!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_set = pd.read_csv(\"cahsi_data_2020/D2.csv\")\n",
    "\n",
    "\n",
    "col = [\n",
    " 'weather_datetime',\n",
    " 'solar_datetime',\n",
    " 'winddirAvg',\n",
    " 'humidityHigh',\n",
    " 'humidityLow',\n",
    " 'humidityAvg',\n",
    " 'heatindexLow',\n",
    " 'heatindexHigh',\n",
    " 'heatindexAvg',\n",
    " 'qcStatus',\n",
    " 'windspeedHigh',\n",
    " 'windgustLow',\n",
    " 'windspeedAvg',\n",
    " 'dewptHigh',\n",
    " 'dewptLow',\n",
    " 'dewptAvg',\n",
    " 'windchillHigh',\n",
    " 'windchillAvg',\n",
    " 'pressureMax',\n",
    " 'pressureMin',\n",
    " 'pressureTrend',\n",
    " 'precipRate',\n",
    " 'precipTotal']\n",
    "\n",
    "testset_features = test_set.drop(col, axis = 1)\n",
    "testset_features = np.array(testset_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the forest's predict method on the test data\n",
    "predictions = rf.predict(testset_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.20963236, 1.20963236, 0.70364704, ..., 6.45444127, 6.45444127,\n",
       "       6.45444127])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step V: Dump predictions into text file for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions:\n",
      " [1.20963236 1.20963236 0.70364704 ... 6.45444127 6.45444127 6.45444127]\n"
     ]
    }
   ],
   "source": [
    "print('Predictions:\\n', predictions) \n",
    "file = open(\"answer.txt\", \"w\") \n",
    "\n",
    "for num in predictions:\n",
    "\n",
    "    content = str(num)\n",
    "    file.write(content)\n",
    "    file.write(\"\\n\")\n",
    "\n",
    "file.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}