03.Naive_Bayse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **나이브 베이즈 모델을 이용한 스팸메일 분류기**\n",
    "Calssification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **1 분류기 Classification**\n",
    "1. **Binary Classification** (이진 분류기) : **True / False 조건을** 구분한다\n",
    "1. **Multiclass Classification** (다변량 분류) : **다양한 클래스간의 조건을** 구분한다\n",
    "1. **Multi-label Classification** (다중 클래스 레이블 분류) : 다중의 클래스간 **겹치는 조건에서** 구분을 한다\n",
    "\n",
    "<img src=\"https://wikidocs.net/images/page/4291/logreg702.PNG\" width=400>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **2 텍스트 분류기 Classification**\n",
    "1. **긍정/ 부정,  긍정/ 중립/ 부정** 분류기\n",
    "1. **뉴스의 토픽** 분류기 (**class 간 중첩되어** 분류가 가능하다)\n",
    "1. **Named Entity Recognition** (개체명 분류기) : ex) Naive Bayse, Support Vector Machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **3  Naive Bayse Classification 개념**\n",
    "1. 확률 기반의 분류기\n",
    "1. **Naive :** 예측을 위한 Token 들이 **Mutually Independent** (상호독립적)을 가정\n",
    "1. **Bayse :** 관찰한 Token이 **클래스 전체 대비, 특정 클래스 속할 확률을 Bayse 기반** 으로 계산\n",
    "\n",
    "> **Naive Bayse 메커니즘**\n",
    "\n",
    "1. 스팸메일과, 정상메일로 구분된 데이터를 사용한다 [download](http://www.aueb.gr/users/ion/data/enron-spam/preprocessed/enron1.tar.gz)\n",
    "1. 단어 **Token을** 대상으로 **스팸여부를** 학습한다.\n",
    "1. Data 추가시 잘못 예측한 결과에 대해 **Laplace Smoothing** 으로 보완한 값을 **Bayse 로 공식을** 수정한다"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **4   Naive Bayse 구현하기**\n",
    "스펨메일 데이터 다운받기 [download](http://www.aueb.gr/users/ion/data/enron-spam/preprocessed/enron1.tar.gz)\n",
    "<br>\n",
    "### **01 enron 메일데이터 살펴보기**\n",
    "1. **Summary.txt** 파일에 저장된 내용 살펴보기\n",
    "1. **정상메일 (3,672개)** 와 **스펨메일 (1,500)개로** 약 1:2의 비율로 구분이 된다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Legitimate\n",
      "----------\n",
      "- Owner: farmer-d\n",
      "- Total number: 3672 emails\n",
      "- Date of first email: 1999-12-10\n",
      "- Date of last email: 2002-01-11\n",
      "- Similars deletion: No\n",
      "- Encoding: No\n",
      "\n",
      "\n",
      "Spam\n",
      "----\n",
      "- Owner: GP\n",
      "- Total number: 1500 emails\n",
      "- Date of first email: 2003-12-18\n",
      "- Date of last email: 2005-09-06\n",
      "- Similars deletion: No\n",
      "- Encoding: No\n",
      "\n",
      "Spam:Legitimate rate = 1:3\n",
      "Total number of emails (legitimate + spam): 5975\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 스팸메일 데이터 Summary\n",
    "with open('./data/enron1/Summary.txt', 'r') as f:\n",
    "    summary = f.read()\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subject: mcmullen gas for 11 / 99\n",
      "jackie ,\n",
      "since the inlet to 3 river plant is shut in on 10 / 19 / 99 ( the last day of\n",
      "flow ) :\n",
      "at what meter is the mcmullen gas being diverted to ?\n",
      "at what meter is hpl buying the residue gas ? ( this is the gas from teco ,\n",
      "vastar , vintage , tejones , and swift )\n",
      "i still see active deals at meter 3405 in path manager for teco , vastar ,\n",
      "vintage , tejones , and swift\n",
      "i also see gas scheduled in pops at meter 3404 and 3405 .\n",
      "please advice . we need to resolve this as soon as possible so settlement\n",
      "can send out payments .\n",
      "thanks\n"
     ]
    }
   ],
   "source": [
    "# ham 폴더에 저장된 메일내용 확인 (정상으로 분류된 메일)\n",
    "file_path = './data/enron1/ham/0007.1999-12-14.farmer.ham.txt'\n",
    "with open(file_path, 'r') as infile:\n",
    "    ham_sample = infile.read()\n",
    "print(ham_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subject: stacey automated system generating 8 k per week parallelogram\n",
      "people are\n",
      "getting rich using this system ! now it ' s your\n",
      "turn !\n",
      "we ' ve\n",
      "cracked the code and will show you . . . .\n",
      "this is the\n",
      "only system that does everything for you , so you can make\n",
      "money\n",
      ". . . . . . . .\n",
      "because your\n",
      "success is . . . completely automated !\n",
      "let me show\n",
      "you how !\n",
      "click\n",
      "here\n",
      "to opt out click here % random _ text\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# spam 폴더에 저장된 메일내용 확인 (스팸으로 분류된 메일)\n",
    "file_path = './data/enron1/spam/0058.2003-12-21.GP.spam.txt'\n",
    "with open(file_path, 'r') as infile:\n",
    "    spam_sample = infile.read()\n",
    "print(spam_sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **02 enron 메일 데이터 분류하기**\n",
    "1. 스펨메일과 정상메일을 레이블을 사용하여 분류한다\n",
    "1. 1 : 스펨메일,  0 : 정상메일\n",
    "1. 분류된 데이터를 전처리 과정을 진행한다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob,os\n",
    "# 정상매일은 0, 스펨매일은 1\n",
    "emails, labels = [], []\n",
    "for no, file_path in enumerate(['./data/enron1/ham/','./data/enron1/spam/']):\n",
    "    for filename in glob.glob(os.path.join(file_path, '*.txt')):\n",
    "        with open(filename, 'r', encoding = \"ISO-8859-1\") as infile:\n",
    "            emails.append(infile.read())\n",
    "            labels.append(no)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **03 enron 메일 데이터 임베딩**\n",
    "1. Chapter 2 에서 진행한 내용을 바탕으로 전처리 작업을 진행한다\n",
    "1. **숫자와 구두점** 제거, **StopWords** 제거, **표제어 원형** 복원\n",
    "1. 정제된 데이터로 **희소벡터 (Sparse Vector)** 로 임베딩 ex) (**row index, feacture/term index**)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fw customer list this one includes more financial counterparties non eol original message from winfree o neal d sent wednesday december am to martin thomas a subject customer list tom attached are eol customer between july and nov broken out by physical and financial the only physical customer i remember non eol are imperial sugar and texas energy i m also checking this list against other non eol deal eric or joe might have done but for now take a look at this o'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import names\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "all_names  = set(names.words())\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "# 표제어 복원작업\n",
    "def clean_text(docs):\n",
    "    cleaned    = [' '.join([lemmatizer.lemmatize(word.lower())\n",
    "                        for word in doc.split()\n",
    "                        if word.isalpha() and word not in all_names])   \n",
    "              for doc in docs]\n",
    "    return cleaned\n",
    "\n",
    "# 사용자 함수를 활용하여 전처리 작업을 진행한다\n",
    "cleaned_emails = clean_text(emails)\n",
    "cleaned_emails[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "모델의 Type: <class 'scipy.sparse.csr.csr_matrix'>\n",
      "임베딩의 크기: (5172, 500)\n",
      "0번문장 내용보기: \n",
      "  (0, 248)\t1\n",
      "  (0, 102)\t1\n",
      "  (0, 125)\t1\n",
      "  (0, 435)\t1\n",
      "  (0, 224)\t1\n",
      "  (0, 30)\t1\n",
      "  (0, 447)\t1\n",
      "  (0, 417)\t1\n",
      "  (0, 104)\t1\n",
      "  (0, 482)\t1\n",
      "  (0, 390)\t1\n",
      "  (0, 265)\t1\n",
      "  (0, 307)\t1\n",
      "  (0, 147)\t2\n",
      "  (0, 241)\t3\n",
      "  (0, 94)\t4\n",
      "  (0, 162)\t1\n",
      "CPU times: user 611 ms, sys: 7.26 ms, total: 618 ms\n",
      "Wall time: 617 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# 출현빈도가 높은 상위 500개의 Token을 대상으로 임베딩 한다\n",
    "# 희소벡터(Sparse Vector)로 변환 : (row index, feacture/term index)\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "cv = CountVectorizer(stop_words=\"english\", max_features=500)\n",
    "term_docs      = cv.fit_transform(cleaned_emails)\n",
    "print(\"모델의 Type: {}\\n임베딩의 크기: {}\\n0번문장 내용보기: \\n{}\".format(\n",
    "    type(term_docs),\n",
    "    term_docs.shape, # 5,172개 문장을 500개 단어로 생성\n",
    "    term_docs [0]))  # 0번 문장의 단어 Vector 목록을 출력"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['able', 'access', 'account', 'accounting', 'act', 'action', 'activity']\n",
      "0 : able\n",
      "162 : fw\n",
      "481 : website\n",
      "357 : read\n",
      "125 : energy\n"
     ]
    }
   ],
   "source": [
    "# cv 모델로 인덱스별 단어 Token 내용보기\n",
    "# feature_mapping = cv.vocabulary_       # dict 로 내용출력 (key:value)\n",
    "\n",
    "print(cv.get_feature_names()[:7])\n",
    "feature_names   = cv.get_feature_names() # List 로 내용출력 (인덱스별 value)\n",
    "for indx in [0, 162, 481, 357, 125]:\n",
    "    print(indx, \":\", feature_names[indx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **04-1 Naive Bayse 학습을 위한 준비작업**\n",
    "모델의 학습을 위한 준비작업으로 데이터를 그룹화 한다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys([0, 1])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[3672, 3673, 3674, 3675, 3676, 3677, 3678, 3679, 3680, 3681]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 레이블을 기준으로 데이터를 그룹화 한다\n",
    "# defaultdict : 스팸여부 0,1 Tag 로 Token Index List 생성\n",
    "def get_label_index(labels):\n",
    "    from collections import defaultdict\n",
    "    label_index = defaultdict(list)\n",
    "    for index, label in enumerate(labels):\n",
    "        label_index[label].append(index)\n",
    "    return label_index\n",
    "\n",
    "# 0 ~ 3600 : 정상메일[0], 3600 ~ 나머지 : 스팸메일[1]\n",
    "label_index = get_label_index(labels)\n",
    "print(label_index.keys())\n",
    "label_index[1][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **04-2 Naive Bayse 위한 사전확률/ 우도값 계산**\n",
    "**사전확률 및 우도값을** 계산하는 함수를 정의한다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.7099767981438515, 1: 0.2900232018561485}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 학습 샘플을 활용하여 사전 확률을 계산 \n",
    "def get_prior(label_index):\n",
    "    \"\"\" Compute prior based on training samples\n",
    "    Args:    label_index (grouped sample indices by class)\n",
    "    Returns: { 단어 key : corresponding prior } \"\"\"\n",
    "    prior = {label: len(index) for label, index in label_index.items()}\n",
    "    total_count = sum(prior.values())\n",
    "    for label in prior:\n",
    "        prior[label] /= float(total_count)\n",
    "    return prior\n",
    "\n",
    "# 위의 인덱스 데이터를 활용하여 사전확률을 계산한다\n",
    "prior = get_prior(label_index)\n",
    "prior"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "우도값 shape : (500,)\n",
      "단어 내용보기 : ['able', 'access', 'account', 'accounting', 'act']\n",
      "우도값 array :\n",
      "[1.08580656e-03 9.57737068e-04 8.79781725e-04 8.46372292e-04\n",
      " 1.00228298e-04 2.39434267e-04 1.97115652e-03 1.34194554e-03\n",
      " 1.84308703e-03 8.35235815e-04 1.08023832e-03 1.03012417e-03\n",
      " 1.67047163e-05 1.50342447e-04 7.62848711e-04 8.40804054e-04\n",
      " 8.12962860e-04 2.22172727e-03 1.99342948e-03 5.01141489e-05]\n"
     ]
    }
   ],
   "source": [
    "# 확률적 유사가능도(최대 가능도 추정)를 계산: 빈도상위 500개의 단어로 조건부 확률 p(feature|spam)을 계산\n",
    "import numpy as np\n",
    "def get_likelihood(term_document_matrix, label_index, smoothing=0):\n",
    "    \"\"\" 훈련 데이터로 우도값 측정\n",
    "    Args:    term_document_matrix, label_index, smoothing        \n",
    "    Returns: { 단어 key, 동시확률 P(feature|class) }\n",
    "    \"\"\"\n",
    "    likelihood = {}\n",
    "    for label, index in label_index.items():\n",
    "        likelihood[label] = term_document_matrix[index, :].sum(axis=0) + smoothing\n",
    "        likelihood[label] = np.asarray(likelihood[label])[0]\n",
    "        total_count       = likelihood[label].sum()\n",
    "        likelihood[label] = likelihood[label] / float(total_count)\n",
    "    return likelihood\n",
    "\n",
    "smoothing  = 1           # 라플라스 스무딩\n",
    "likelihood = get_likelihood(term_docs, label_index, smoothing)\n",
    "print(\"우도값 shape : {}\\n단어 내용보기 : {}\\n우도값 array :\\n{}\".format(\n",
    "    likelihood[0].shape, # 0번 레이블일 때 단어별 우도값 계산\n",
    "    feature_names[:5],   # 인덱스별 단어 확인\n",
    "    likelihood[0][:20])) # 0번 레이블의 단어별 우도값 샘플 [:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **04-3 자연 Log 를 활용한 예측함수 구현하기**\n",
    "- 앞에서 측정한 **사전확률과 및 우도값을** 활용하여 예측함수를 정의 합니다\n",
    "- 단어들의 확률을 합치기 위해, **Log()** 로 변환 후 **경우의 수를 모두 합칩니다**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OverFlow가 발생 가능하므로, 데이터를 Log() 자연로그로 변환 후 덧셈 계산,\n",
    "# 계산이 끝난 뒤, 로그의 역함수 (exp()) 를 활용하여 실수로 변환한다\n",
    "def get_posterior(term_document_matrix, prior, likelihood):\n",
    "    \"\"\" 사전확률과 유사가능도를 바탕으로 샘플 데이터의 사후확률을 계산\n",
    "    Args:\n",
    "        term_document_matrix (sparse matrix)\n",
    "        prior      { 단어 Key : 사전확률 }\n",
    "        likelihood { 단어 Key : 조건부 확률 }\n",
    "    Returns:       { 단어 Key : 관련 사후 확률값 }\n",
    "    \"\"\"\n",
    "    # 확률의 연산시 log() 로 변환한 후 합친다\n",
    "    num_docs, posteriors = term_document_matrix.shape[0], []\n",
    "    for i in range(num_docs):\n",
    "        # 사후확률 : 사전확률 X 유사가능도(최대 가능도 추정량)\n",
    "        posterior = {key: np.log(prior_label) for key, prior_label in prior.items()}\n",
    "        for label, likelihood_label in likelihood.items():\n",
    "            term_document_vector = term_document_matrix.getrow(i)\n",
    "            counts  = term_document_vector.data\n",
    "            indices = term_document_vector.indices\n",
    "            for count, index in zip(counts, indices):\n",
    "                posterior[label] += np.log(likelihood_label[index]) * count\n",
    "        # exp(-1000):exp(-999) 는 분모가 0이 되는 문제가 발생\n",
    "        # 하지만 exp(0):exp(1)과 동치가 된다.\n",
    "        min_log_posterior = min(posterior.values())\n",
    "        for label in posterior:\n",
    "            try:    posterior[label] = np.exp(posterior[label] - min_log_posterior)\n",
    "            except: posterior[label] = float('inf') # 값이 너무 클때\n",
    "        # 전체 합이 1이 되도록 정규화\n",
    "        sum_posterior = sum(posterior.values())\n",
    "        for label in posterior:\n",
    "            if posterior[label] == float('inf'): posterior[label] = 1.0\n",
    "            else: posterior[label] /= sum_posterior\n",
    "        posteriors.append(posterior.copy())\n",
    "    return posteriors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{0: 0.9815828838777807, 1: 0.018417116122219333},\n",
      " {0: 1.5274461154428757e-06, 1: 0.9999984725538845}]\n"
     ]
    }
   ],
   "source": [
    "# 테스트 메일을 사용하여 알고리즘을 검증\n",
    "emails_test = [\n",
    "    '''Subject: flat screens hello ,\n",
    "    please call or contact regarding the other flat screens requested .\n",
    "    trisha tlapek - eb 3132 b michael sergeev - eb 3132 a\n",
    "    also the sun blocker that was taken away from eb 3131 a .\n",
    "    trisha should two monitors also michael .thanks kevin moore''',\n",
    "    \n",
    "    '''Subject: having problems in bed ? we can help !\n",
    "    cialis allows men to enjoy a fully normal sex life without having to plan the sexual act .\n",
    "    if we let things terrify us , life will not be worth living .\n",
    "    brevity is the soul of lingerie . suspicion always haunts the guilty mind .''']\n",
    "\n",
    "cleaned_test   = clean_text(emails_test)\n",
    "term_docs_test = cv.transform(cleaned_test)\n",
    "posterior      = get_posterior(term_docs_test, prior, likelihood)\n",
    "from pprint import pprint\n",
    "pprint(posterior)\n",
    "# 검증결과 0번 메일은 0.98로 정상, 1번 메일은 0.99로 스펨에 해당"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **04-4 학습을 위해 Train / Test 데이터를 나눈다**\n",
    "scikit-learn 모듈 **train_test_split** 을 사용한다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({0: 3672, 1: 1500})"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "Counter(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train 'email':3,465, 'label':3,465\n",
      "Test  'email':1,707, 'label':1,707\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(\n",
    "    cleaned_emails,  # X_train, X_test 로 추출\n",
    "    labels,          # Y_train, Y_test 로 추출\n",
    "    test_size    = 0.33, \n",
    "    random_state = 42)\n",
    "print(\"Train 'email':{:,}, 'label':{:,}\\nTest  'email':{:,}, 'label':{:,}\".format(\n",
    "    len(X_train), len(Y_train), len(X_test), len(Y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/markbaum/Python/python/lib/python3.6/site-packages/ipykernel_launcher.py:26: RuntimeWarning: overflow encountered in exp\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1,707 개의 테스트 데이터(Y_test)의 정확도는: 91.6 %\n"
     ]
    }
   ],
   "source": [
    "# 데이터 Set의 사후 확률을 예측한다\n",
    "term_docs_train = cv.fit_transform(X_train)\n",
    "label_index     = get_label_index(Y_train)\n",
    "prior           = get_prior(label_index)\n",
    "likelihood      = get_likelihood(term_docs_train, label_index, smoothing)\n",
    "\n",
    "# Test / 신규 데이터 Set의 사후확률을 예측한다\n",
    "term_docs_test = cv.transform(X_test)\n",
    "posterior = get_posterior(term_docs_test, prior, likelihood)\n",
    "correct   = 0.0\n",
    "\n",
    "for pred, actual in zip(posterior, Y_test):\n",
    "    if actual == 1:\n",
    "        if pred[1] >= 0.5: correct += 1\n",
    "    elif pred[0] > 0.5: correct += 1\n",
    "\n",
    "# dtype 을 128 이상으로 지정할 것\n",
    "# https://stackoverflow.com/questions/40726490/overflow-error-in-pythons-numpy-exp-function/40726641\n",
    "print('{:,} 개의 테스트 데이터(Y_test)의 정확도는: {:.1f} %'.format(\n",
    "    len(Y_test), correct/len(Y_test)*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/>\n",
    "\n",
    "## **5 Sklearn 을 활용한 Naive Bayse 구현하기**\n",
    "- 위에서 복잡한 과정을 sklearn으로 실습 합니다\n",
    "- nltk 모듈을 활용한 예제 [nltk_tutorial](https://nbviewer.jupyter.org/github/YongBeomKim/nltk_tutorial/blob/master/ipython/03-2.Bayse.ipynb)\n",
    "### **01 데이터 전처리 및 모델학습**\n",
    "모델을 학습한 뒤 정확도를 측정한다\n",
    "```python\n",
    "# cleaned_emails[0] : 전처리된 텍스트 List\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "cv = CountVectorizer(stop_words=\"english\", max_features=500)\n",
    "term_docs_test = cv.transform(cleaned_test)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9.99808489e-01, 1.91511166e-04],\n",
       "       [9.99999772e-01, 2.28176513e-07],\n",
       "       [9.99999223e-01, 7.77402015e-07],\n",
       "       [9.99999724e-01, 2.76311984e-07],\n",
       "       [9.98447799e-01, 1.55220148e-03],\n",
       "       [1.00000000e+00, 2.17331050e-15]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "clf = MultinomialNB(alpha     = 1.0,  # 라플라스 Smoothing 값\n",
    "                    fit_prior = True) # Data Set로 학습된 사전확률 사용\n",
    "clf.fit(term_docs_train, Y_train)\n",
    "prediction_prob = clf.predict_proba(term_docs_test)\n",
    "prediction_prob[0:6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 1, 0, 0])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 예측한 클래스 값을 바로 계산하여 출력한다\n",
    "# 역치값은 0.5로 0.5보다 크면 1, 작으면 0을 출력\n",
    "prediction = clf.predict(term_docs_test)\n",
    "prediction[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy using MultinomialNB is: 91.6%\n"
     ]
    }
   ],
   "source": [
    "# test 값을 활용하여 모델의 정확도 측정 \n",
    "accuracy = clf.score(term_docs_test, Y_test)\n",
    "print('The accuracy using MultinomialNB is: {0:.1f}%'.format(accuracy*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **02 분류기의 성능 평가**\n",
    "**혼동행렬(Confusion Matrix) 분할표로** 예측값을 테스트하여 출력한다\n",
    "\n",
    "<img src=\"https://docs.microsoft.com/ko-kr/azure/machine-learning/studio/media/evaluate-model-performance/6a.png\" align=\"left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1138,   91],\n",
       "       [  52,  426]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 혼동행렬을 계산\n",
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(Y_test, prediction, labels=[0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precesion(정밀도) : 0.824\n",
      "Recall(재현율) : 0.8912\n",
      "f1 score (1) : 0.8563 \n",
      "f1 score (0) : 0.9409\n"
     ]
    }
   ],
   "source": [
    "# f1 Score 를 측정하여 정밀도, 재연율을 계산\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score\n",
    "print(\"\"\"Precesion(정밀도) : {:.4}\\nRecall(재현율) : {:.4}\n",
    "f1 score (1) : {:.4} \\nf1 score (0) : {:.4}\"\"\".format(\n",
    "    precision_score(Y_test, prediction, pos_label=1),\n",
    "    recall_score(Y_test, prediction, pos_label=1),\n",
    "    f1_score(Y_test, prediction, pos_label=1),\n",
    "    f1_score(Y_test, prediction, pos_label=0)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.93      0.94      1229\n",
      "           1       0.82      0.89      0.86       478\n",
      "\n",
      "   micro avg       0.92      0.92      0.92      1707\n",
      "   macro avg       0.89      0.91      0.90      1707\n",
      "weighted avg       0.92      0.92      0.92      1707\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 위 내용을 한꺼번에 실행해본다\n",
    "from sklearn.metrics import classification_report\n",
    "report = classification_report(Y_test, prediction)\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **03 분류기의 성능 평가**\n",
    "1. **정확도**(훈련데이터 적합도) 와 **재현율**(일반화 정도)이 **모두 높은 경우가 없기** 때문에 f1-score를 측정한다\n",
    "1. 하지만 모델의 **평균값과,** 모델의 **f1-score** 둘 다 높은 모델은 없으므로 별도 기준이 필요\n",
    "1. 대표적인 대안으로 **ROC (Receiver Operation Characteristic), AUC (Area Under the Curve)** 가 있다\n",
    "1. 이번 예제에서는 **ROC**를 그려보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.8 ms, sys: 0 ns, total: 2.8 ms\n",
      "Wall time: 2.77 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# ROC Curve 값들을 계산합니다\n",
    "pos_prob   = prediction_prob[:, 1]\n",
    "thresholds = np.arange(0.0, 1.2, 0.1)\n",
    "true_pos   = [0]*len(thresholds) \n",
    "false_pos  = [0]*len(thresholds)\n",
    "\n",
    "for pred, y in zip(pos_prob, Y_test):\n",
    "    for i, threshold in enumerate(thresholds):\n",
    "        if pred >= threshold:\n",
    "            if y == 1: true_pos[i] += 1\n",
    "            else:      false_pos[i] += 1\n",
    "        else: break\n",
    "\n",
    "# 임계치를 설정하기 위해 양성비율과, 음성 비율을 계산한다\n",
    "# 양성 테스트 샘플이 516개, 음성 테스트 샘플이 1,191개 이다\n",
    "true_pos_rate  = [tp / 516.0  for tp in true_pos]\n",
    "false_pos_rate = [fp / 1191.0 for fp in false_pos]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd4VGX2wPHvSU/oJIBIF5AiIGhEFAUUBQRs2EDFhgURFfGH6KJi3VVUFKUE27rKuhZ2VQREUUFsKKiAgoA0IUiH0BNSzu+PewNjTCaTkJk7k5zP8+SZuf3Mzcyced/33vcVVcUYY4wpSpTXARhjjAlvliiMMcb4ZYnCGGOMX5YojDHG+GWJwhhjjF+WKIwxxvhlicIETESuEpFPvI4jnIjIPhE5zoPjNhYRFZGYUB87GERkqYh0K8V29p4MAUsUEUpE1onIQfeLarOIvCYilYN5TFX9t6r2COYxfInI6SLyuYjsFZHdIvKhiLQO1fELiWeuiNzoO09VK6vqmiAd73gReVdEtruvf4mIDBeR6GAcr7TchNXsaPahqieo6txijvOX5Bjq92RFZYkisp2vqpWB9kAH4D6P4ymVwn4Vi8hpwCfAB8CxQBNgMfB1MH7Bh9svcxFpCnwHbADaqmo14DIgFahSxsfy7LWH23k3RVBV+4vAP2AdcI7P9Bhghs90PPA0sB7YAqQBiT7LLwQWAXuA1UAvd3414BVgE7AReAyIdpddB3zlPp8EPF0gpg+A4e7zY4H/AtuAtcAdPus9BEwFprjHv7GQ1/clMLGQ+R8Br7vPuwHpwN+A7e45uSqQc+Cz7UhgM/AGUAOY7sa8y31e313/cSAXyAT2AePd+Qo0c5+/BkwAZgB7cb7om/rE0wNYAewGJgJfFPba3XWn+P4/C1ne2D32te7r2w6M8lneEfgWyHD/l+OBOJ/lCtwG/AasdeeNw0lMe4AfgDN91o92z/Nq97X9ADQA5rn72u+elyvc9fvivL8ygG+AdgXeuyOBJUAWEIPP+9mNfaEbxxZgrDt/vXusfe7fafi8J911TgBmAzvdbf/m9We1PPx5HoD9lfIf9+cPVn3gZ2Ccz/JngWlATZxfoB8C/3CXdXS/rM7FKVXWA1q6y94DJgOVgNrA98At7rLDH0qgi/ulIu50DeAgToKIcr9IHgTigOOANUBPd92HgGzgInfdxAKvLQnnS/msQl739cAm93k3IAcYi5MUurpfWC0COAf52z7pbpsIJAOXuMevArwLvO9z7LkU+GLnr4lih3t+Y4B/A2+5y1LcL75+7rI73XNQVKLYDFzv5//f2D32S27sJ+J86bZyl58MdHKP1Rj4FRhWIO7Z7rnJT55Xu+cgBrjbjSHBXTYC5z3WAhD3eMkFz4E73QHYCpyKk2CuxXm/xvu8dxfhJJpEn3n57+dvgYHu88pApwKvOcbnWNdx5D1ZBScp3g0kuNOnev1ZLQ9/ngdgf6X8xzkfrH04v+4U+Ayo7i4TnC9M31+zp3Hkl+Nk4NlC9lnH/bLxLXkMAOa4z30/lILzC6+LO30T8Ln7/FRgfYF93wf8033+EDDPz2ur776mloUs6wVku8+74XzZV/JZ/g7wQADnoBtwKP+LsIg42gO7fKbnUnyieNlnWW9gufv8GuBbn2WCk2iLShTZuKW8Ipbnf2nW95n3PdC/iPWHAe8ViPvsYt5ju4AT3ecrgAuLWK9gopgEPFpgnRVAV5/37g2FvJ/zE8U84GEgpYjXXFSiGAD8FMzPXUX9s/rByHaRqn4qIl2BN3F+tWYAtXB+Ff8gIvnrCs6vO3B+yc0sZH+NgFhgk892UThfaH+iqioib+F8OOcBV+JUl+Tv51gRyfDZJBqnOinfX/bpYxeQB9QFlhdYVhenmuXwuqq632f6d5xSTXHnAGCbqmYeXiiShFMK6YVTQgKoIiLRqprrJ15fm32eH8D5RYwb0+HX7J6/dD/72YHzWkt1PBE5HqeklYpzHmJwSnm+/vQ/EJH/Awa5sSpQFec9Bc57ZnUA8YDz/79WRG73mRfn7rfQYxcwCHgEWC4ia4GHVXV6AMctSYymBKwxuxxQ1S9wfs0+7c7ajlMNdIKqVnf/qqnT8A3Oh7RpIbvagFOiSPHZrqqqnlDEof8DXCoijXBKEf/12c9an31UV9UqqtrbN2w/r2c/TvXDZYUsvhyn9JSvhohU8pluCPwRwDkoLIa7capWTlXVqjjVa+AkGL8xB2ATTknJ2aGTveoXvTqf4lSDldYknCTb3H0tf+PI68h3+PWIyJnAPTjnt4aqVsepnszfpqj3TGE2AI8X+P8nqep/Cjt2Qar6m6oOwKn6fBKY6v6Pizv/G3CqOU0Zs0RRfjwHnCsiJ6pqHk7d9bMiUhtAROqJSE933VeA60Wku4hEuctaquomnCuNnhGRqu6ypm6J5S9U9SecL+SXgY9VNb8E8T2wV0RGikiiiESLSBsROaUEr+denF+ld4hIFRGpISKP4VQfPVxg3YdFJM79susLvBvAOShMFZzkkiEiNYHRBZZvofRfRDOAtiJykXulz23AMX7WHw2cLiJPicgxbvzNRGSKiFQP4HhVcNpE9olIS+DWANbPwWnIjxGRB3FKFPleBh4VkebiaCciye6yguflJWCwiJzqrltJRPqISEBXa4nI1SJSy/0f5r+n8tzY8ij6fzAdqCsiw0Qk3n3fnBrIMY1/lijKCVXdBryO04AMzlUlq4D5IrIH5xdqC3fd73EahZ/F+dX4BU51ATh16XHAMpwqoKn4rwJ5EzjHfcyPJRfnC7s9zhVP+cmkWglez1dAT5zG3004VUodgDNU9TefVTe7cf6B03g8WFXzq6uKPAdFeA6nYXg7MB+YVWD5OJwS1C4ReT7Q1+K+nu04JaQxONVKrXGu7MkqYv3VOEmxMbBURHbjlNgW4rRLFef/cKoD9+J8cb9dzPof47zelTjnOpM/Vw+NxWn/+QQnAb2Cc67AaXP6l4hkiMjlqroQp81qPM7/ZhVOW0KgeuG85n0457y/qh5U1QM4V5997R6rk+9GqroX5wKN83HeF78BZ5XguKYI+VesGBNx3Dt5p6iqvyqcsCQiUTiX516lqnO8jscYf6xEYUyIiEhPEakuIvEcaTOY73FYxhTLEoUxoXMazlU523GqRy5S1YPehmRM8azqyRhjjF9WojDGGONXxN1wl5KSoo0bN/Y6DGOMiSg//PDDdlWtVZptIy5RNG7cmIULF3odhjHGRBQR+b2021rVkzHGGL8sURhjjPHLEoUxxhi/LFEYY4zxyxKFMcYYvyxRGGOM8StoiUJEXhWRrSLySxHLRUSeF5FVIrJERE4KVizGGGNKL5glitdwugsuynlAc/fvZpyBVowxxpSxQ4cCHaCxcEG74U5V54lIYz+rXAi8rk5nU/PdXjXruoPnGGOMKQMjRnzCTz9tLn5FP7xso6jHnwdGSXfn/YWI3CwiC0Vk4bZt20ISnDHGlAdt2tTmyy/XH9U+IqIxW1VfVNVUVU2tVatUXZUYY0yFsGzZNqZMWXJ4+pprTmTFiqFHtU8v+3raCDTwma7vzjPGGFNCBw5k89hj83jqqW+IjhY6dapPs2Y1EREaNw5kmPWieZkopgFDReQt4FRgt7VPGGNMyX300W/cdttM1q7NAGDQoJNJTk4sZqvABS1RiMh/gG5AioikA6OBWABVTQNmAr1xBl4/AFwfrFiMMaY82rhxD8OGfczUqcsAaNeuDmlpfTjttAbFbFkywbzqaUAxyxW4LVjHN8aY8u6222bywQcrSEqK5ZFHunHnnZ2IiSn7pueIG4/CGGMqspycvMPJ4MknzyE2NppnnulBw4bVgnbMiLjqyRhjKrrduzO5/faZ9OnzJk6FDLRokcK7714W1CQBVqIwxpiwpqq8++4yhg2bxaZN+4iOFhYt2kyHDnVDFoMlCmOMCVOrV+9k6NCPmDVrFQCnnVaftLS+tGtXJ6RxWKIwxpgw9PTT3/DAA3PIzMyhevUEnnzyHG688SSioiTksViiMMaYMHTgQDaZmTkMHNiOp5/uQe3alTyLxRKFMcaEgW3b9rNixQ7OOKMhACNHdqZbt8Z06dLI48jsqidjjPFUXp7y8ss/0qLFePr1e5udOw8CEB8fExZJAqxEUb6pAlr8YyDrlGRdv8sKiSvcYizt/sr6nIfba46k/12pjknIj5udncvu3Zn0PZTD+XdAfFwUVd58EoQyPh9HxxJFoDYvgLnD4eD24H4AICzeGMaY4IsFUhKABJ+ZWR4F44clikDs+R3e6wsHtnodSSkJiPh/LHIZAa5XwnXKen9lekwq0GvN/yP0r8Pr/Xn4mgde8z5z5q4DhGuuOZGR955JtWqJwX3N/1f6lgZLFMXJy4H3L3CSRMNz4KznIufDcPhLzxgTTgYNu4Cl6z9m0qQ+nHpqfa/DKZYliuJs+RG2LYHK9eD8dyChhtcRGWMiSE5OHi+88B3r1mUwbtx5AHTr1piFC2/25J6I0rBEUZytPzqPDc+2JGGMKZHvv9/ILbdMZ9EiZ8zqm28+mRNOqA0QMUkC7PLY4m39yXms3cHbOIwxESMjI5MhQ2bQqdPLLFq0mUaNqvHhhwMOJ4lIYyWK4mxxSxS1T/I2DmNMRHjrrV8YNmwWW7bsJyYmirvvPo0HHuhCpUpxXodWapYo/MnNhu0/O89rt/c2FmNMRPjkk9Vs2bKfzp0bMGlSH9q2DW0HfsFgicKfncshNwuqN4X44Pb3boyJTFlZOWzcuJfjjnPaMMeMOZczz2zItde2j6h2CH+sjcKf/IZsa58wxhTi88/X0q5dGn36vMmhQ7kApKQkcf31HcpNkgBLFP5ZQ7YxphBbtuxj4MD36N79dVau3AFAevoej6MKHqt68scaso0xPvLylJde+oF77/2MjIxMEhJiuP/+MxkxojNxcdFehxc0liiKonmwbZHz3EoUxhjg4ovfZtq0FQD07NmUCRN607RpTY+jCj6reipKxho4tBcqHwuVIv+qBWPM0evXryXHHFOZt9++lI8+uqpCJAmwEkXRNsxxHq00YUyFNW3aCtLT9zBkyCkAXHPNifTr14oqVeI9jiy0LFEU5tA++PYh5/nxl3kaijEm9Nav380dd3zEBx+sID4+ml69mnHccTUQkQqXJMASReEWPAn7/oBjToHWA72OxhgTItnZuTz//HeMHj2X/fuzqVIljsceO5tGjSr2fVSWKAqzcqrzeOaTINaMY0xFMH9+OrfcMp0lS7YAcNllrXn22Z7Uq1fV48i8Z4mioNxsyFgFCNQ91etojDEh8sADc1iyZAtNmlRn/Pje9O7d3OuQwoYlioJ2r3UGK6raCGKTvI7GGBMkqsrevYeoWtVpcxg//jxef30xo0Z1ISkp1uPowovVqxS04xfnsUYLb+MwxgTNihXbOeecN+jX721UnTHmW7RI4fHHu1uSKISVKAr67X/OY/0zvY3DGFPmMjNz+Mc/vuSJJ77m0KFckpMTWbcugyZNbFAyfyxRFPT7bOfx+Mu9jcMYU6Zmz17NkCEzWbVqJwA33NCeMWPOJTnZqpiLE9SqJxHpJSIrRGSViNxbyPKGIjJHRH4SkSUi0juY8RQrLwcObAMEqh/naSjGmLKhqtxwwwf06DGFVat20rp1LebNu45XXrnQkkSAglaiEJFoYAJwLpAOLBCRaaq6zGe1+4F3VHWSiLQGZgKNgxVTsQ5uBxQSa0GUFbaMKQ9EhMaNq5OYGMODD3Zl+PDTynUHfsEQzG/DjsAqVV0DICJvARcCvolCgfyLlKsBfwQxnuLt+d15TIrMcW2NMY5FizazadNezjvPucR15MjODBzYztoiSimYVU/1gA0+0+nuPF8PAVeLSDpOaeL2wnYkIjeLyEIRWbht27ZgxOpY97HzeOzpwTuGMSZo9u7NYvjwjzn55Be59tr32bnzIADx8TGWJI6C15fHDgBeU9X6QG/gDZG/3gqtqi+qaqqqptaqVSt40az+0Hlsen7wjmGMKXOqynvv/Urr1hN59tn5AFx5ZVtiY73+iisfgln1tBFo4DNd353naxDQC0BVvxWRBCAF2BrEuAq3bxNsWQgxCdCwe8gPb4wpnd9/z2Do0I+YPn0lAKmpxzJ5cl9OOqmux5GVH8FMtwuA5iLSRETigP7AtALrrAe6A4hIKyABCGLdkh9rZjiPDc+xO7KNiRCqyiWXvMP06SupWjWe8ePPY/78QZYkyljQShSqmiMiQ4GPgWjgVVVdKiKPAAtVdRpwN/CSiNyF07B9nebfJhlqa6zayZhIkZenREUJIsLTT/cgLW0hzz7bk7p1q3gdWrkkXn0vl1ZqaqouXLiwbHeqChNqQNZuuGk9VG1Q/DbGmJDbseMA9977KQAvvXSBx9FEFhH5QVVTS7OttfSA0xFg1m6odIwlCWPCkKryr38tomXLCbz88k+8/voS0tP3eB1WhWF3lQFs/dF5rH2St3EYY/7i11+3ceutM/jiC+c+p27dGjNpUh/q17dxIkLFEgXAFjdR1LFEYUy4UFUefHAOTz75NdnZeaSkJPHMMz0YOLAdIuJ1eBWKJQrwKVF08DYOY8xhIsLGjXvJzs7jpptO4oknzqFmzUSvw6qQLFGoHilRWNWTMZ7644+9bN9+gHbt6gAwZsy5DBrUgc6dG3ocWcVmjdn7NsLBbZBQwxnVzhgTcrm5eYwf/z2tWk2gf/+pHDqUC0BKSpIliTBgJYqti5zH2h3A6j2NCbkff9zELbdMZ+FCp0/QLl0asWdPFikpduNruAgoUbh3VjdU1VVBjif0drid2Sa38TYOYyqYPXuyeOCBzxk/fgF5eUr9+lV5/vleXHRRS2usDjPFJgoR6QOMBeKAJiLSHhitqhcHO7iQ2LHUeUw5wds4jKlAVJUuXf7J4sVbiI4Whg/vxEMPdaNKlXivQzOFCKSN4hHgVCADQFUXAc2CGVRI5Zcoarb2Ng5jKhAR4a67OtGxYz0WLryZZ57paUkijAVS9ZStqhkFioKR1e9HUTQPdv7qPE9u5W0sxpRjhw7lMnbst0RHCyNGdAbgmmtO5Oqr2xEdbdfUhLtAEsWvInI5ECUiTYA7gPnBDStE9m6A7P2QVAcSk72Oxphy6csvf2fw4BksW7aN+PhorrnmROrUqYyIEB1tbRGRIJBUPhQ4GcgD/gdkAXcGM6iQOdyQbdVOxpS17dsPcMMNH9Cly2ssW7aN5s1rMn36ldSpU9nr0EwJBVKi6KmqI4GR+TNEpB9O0ohsliiMKXOqymuvLWLEiNns2HGQuLho7rvvDO699wwSEuyK/EgUSIni/kLmjSrrQDxhicKYoJgy5Wd27DjI2Wc3YcmSwTz0UDdLEhGsyP+ciPTEGaa0noiM9VlUFacaKvLlXxqbbJfGGnM0DhzIZvfuTOrWrYKIMHFibxYs+IOrrmpr90SUA/5S/FbgFyATWOozfy9wbzCDCglVK1EYUwY++ug3brttJscdV4PZswciIrRokUKLFileh2bKSJGJQlV/An4SkX+ramYIYwqNfRvh0F5ITIGkWl5HY0zE2bhxD8OGfczUqc4PripV4tmx46B1vVEOBVJpWE9EHgdaAwn5M1X1+KBFFQpWmjCmVHJz85gwYQH33/85e/ceolKlWB555CzuuONUYmLsnojyKJBE8RrwGPA0cB5wPeXhhjtLFMaUWF6e0rXra3z99QYALrqoJePG9aJhw2oeR2aCKZD0n6SqHwOo6mpVvR8nYUQ267rDmBKLihJ69GhKgwZV+eCD/rz33hWWJCqAQEoUWSISBawWkcHARqBKcMMKAStRGFMsVeWdd5YSExPFJZc4n5WRIzszfPhpVK4c53F0JlQCSRR3AZVwuu54HKgG3BDMoIJO1XqNNaYYq1fvZMiQmXzyyWpq1Uri7LObUKNGIvHxMcRb/30VSrGJQlW/c5/uBQYCiEi9YAYVdPs3Q1aGM6pdUh2vozEmrGRl5fDUU9/w+ONfkpmZQ40aCTz++NlUq5ZQ/MamXPKbKETkFKAe8JWqbheRE3C68jgbqB+C+ILDt33CbgYy5rC5c9dx660zWL58OwADB7bj6ad7ULt2JY8jM14qsjFbRP4B/Bu4CpglIg8Bc4DFgF0aa0w5k5ubx5AhTpJo0SKZzz+/htdfv9iShPFborgQOFFVD4pITWAD0FZV14QmtCDaaYnCGHAud83MzCEpKZbo6CgmTerDvHm/c889nYmPt76ZjMPfOyFTVQ8CqOpOEVlZLpIEWInCGODnn7cwePAMWrZM5pVXLgSga9fGdO3a2NvATNjxlyiOE5H8rsQFZ7zsw12Lq2q/oEYWTJYoTAW2f/8hHnnkC8aOnU9OTh5r1+5i166D1KiR6HVoJkz5SxSXFJgeH8xAQubAVji4HeKqQuXIvnjLmJL68MMVDB36EevX70YEhgxJ5fHHu1O9ul3RZIrmr1PAz0IZSMj4libsiidTQeTk5HHFFVP53/+cMeLbtz+GyZP70rGj/Vgyxat4rVVW7WQqoJiYKKpVi6dy5TgeffQshg7taB34mYAF9Z0iIr1EZIWIrBKRQsewEJHLRWSZiCwVkTeDGQ9gicJUGN99l85336Ufnn7qqXP59dfbGDaskyUJUyIBlyhEJF5Vs0qwfjQwATgXSAcWiMg0VV3ms05z4D6gs6ruEpHagYdeSpYoTDmXkZHJffd9yuTJP9CyZQqLFg0mLi6a5GQbJ8KUTrE/K0Sko4j8DPzmTp8oIi8EsO+OwCpVXaOqh4C3cO7N8HUTMEFVdwGo6tYSRV8alihMOaWqvPnmz7RsOZ60tB+Ijo7iggtakJtbPkYuNt4JpETxPNAXeB9AVReLyFkBbFcP5ya9fOnAqQXWOR5ARL4GooGHVHVWAPsunYM74MAWiK0EVRoE7TDGhNpvv+1gyJCZfPqpc6tT584NSEvrS5s2wS+km/IvkEQRpaq/FxggPbcMj98c6IbTd9Q8EWmrqhm+K4nIzcDNAA0bNiz90f50xZPV0ZryITs7l7PPfp309D3UrJnImDHncP31HYiKsqv6TNkIJFFsEJGOgLrtDrcDKwPYbiPg+7O9vjvPVzrwnapmA2tFZCVO4ljgu5Kqvgi8CJCamlr60fWs2smUI6qKiBAbG83jj5/NnDnrGDPmHGrVsr6ZTNkK5Gf1rcBwoCGwBejkzivOAqC5iDQRkTigPzCtwDrv45QmEJEUnKqo4HUTYqPamXJgy5Z9DBz4Ho89Nu/wvGuuOZF//vNCSxImKAIpUeSoav+S7lhVc0RkKPAxTvvDq6q6VEQeARaq6jR3WQ8RWYZTnTVCVXeU9FgBsxKFiWB5ecpLL/3Avfd+RkZGJtWrJzBsWCeqVLFRhExwBZIoFojICuBt4H+qujfQnavqTGBmgXkP+jxXnNLK8ED3eVSs11gToRYv3szgwTOYP9+5L6JXr2ZMmNDbkoQJiUBGuGsqIqfjVB09LCKLgLdU9a2gR1eWMjNg3x8QkwhVG3kdjTEByc7O5b77PuO55+aTm6vUrVuZceN6cemlrRHrgsaESECX/qjqN6p6B3ASsAdnQKPIstPp44aaLSEq2ttYjAlQTEwUP/20mbw85fbbO/Lrr7dx2WUnWJIwIVVsiUJEKuPcKNcfaAV8AJwe5LjK3valzmPyCd7GYUwx1q/fTW5uHk2a1EBESEvrw+7dWaSmHut1aKaCCqSN4hfgQ2CMqn4Z5HiCx9onTJjLzs5l3LjvGD16LqedVp/ZswciIjRvnux1aKaCCyRRHKeqkd8HgF3xZMLYt99uYPDgGSxZsgWAmjUTOXAgm0qV4jyOzBg/iUJEnlHVu4H/ishfbnKLuBHuLFGYMLRr10HuvfdTXnzxRwCaNKnOhAm9Oe+85h5HZswR/koUb7uPkT+yXdYe2LsBouOhWhOvozEGgKysHNq3n8z69buJjY1ixIjTGTWqC0lJsV6HZsyf+Bvh7nv3aStV/VOycG+ki5wR8HYudx5rtoCoijdWkwlP8fExDBrUgc8+W8ukSX1o3bqW1yEZU6hALo+9oZB5g8o6kKCyrjtMGMjMzGH06Dm8+ebPh+f97W9nMnfutZYkTFjz10ZxBc4lsU1E5H8+i6oAGYVvFaZ2uJfGptilscYbs2evZsiQmaxatZPatStx8cUtSUyMtZHmTETwVw/zPbADp9fXCT7z9wI/BTOoMmcN2cYjmzfvY/jwj/nPf34B4IQTapGW1pfERGuHMJHDXxvFWmAt8GnowgkSq3oyIZabm8fkyT/wt799xu7dWSQmxjB6dFfuuus04uKsZwATWfxVPX2hql1FZBfge3ms4PTnVzPo0ZWF7P2wZx1ExUL1pl5HYyqI3FzlhRe+Z/fuLHr3bs748efRpEkNr8MyplT8VT3lD3eaEopAgib/iqcax0O0FfdN8Ozdm0VurlK9egJxcdG89NL5bNmyj379WlnfTCaiFdmS5nM3dgMgWlVzgdOAW4DIGR3F2idMkKkq//vfr7RqNYG77/748PwzzmjIJZdYL68m8gVyycX7OMOgNgX+iTNU6ZtBjaosWaIwQbRuXQYXXPAWl1zyDhs37uWXX7aRmZnjdVjGlKlAEkWeO6Z1P+AFVb0LqBfcsMrQ4URhl8aaspOdncuTT35F69YTmD59JVWrxjN+/Hl8880NJCTYTZ2mfAloKFQRuQwYCFzkzoucyv78eyisRGHKyIED2XTq9DI//7wVgP792zB2bA/q1q3icWTGBEcgieIGYAhON+NrRKQJ8J/ghlVGsg9CxhqQaKhhnayZspGUFEtq6rEcOJDNxIl96NHDrqYz5VsgQ6H+IiJ3AM1EpCWwSlUfD35oZWDXCkCdJBFt3TWb0lFVXn99MU2b1uSMMxoC8OyzPYmLi7Yb50yFEMgId2cCbwAbce6hOEZEBqrq18EO7qhZQ7Y5Sr/+uo1bb53BF1/8TqtWKSxaNJi4uGiqVUvwOjRjQiaQqqdngd6qugxARFrhJI7UYAZWJixRmFI6eDCbxx//kjFjviY7O49atZK4774ziI21vplMxRNIoojLTxIAqvqriERGPY513WFKYdasVdxY6kbmAAAZDElEQVR220zWrNkFwE03ncQTT5xDzZqJHkdmjDcCSRQ/ikgaMMWdvopI6RTQShSmhPbtO8TAge+xffsB2rSpTVpaHzp3buh1WMZ4KpBEMRi4A7jHnf4SeCFoEZWVnCzIWAUS5QxYZEwRcnPzyMtTYmOjqVw5jnHjepGevoe77upEbKx14GeM30QhIm2BpsB7qjomNCGVkV0rQXOdK55irOHRFO6HH/7gllumc+GFLXjgga4AXHllW4+jMia8FNkyJyJ/w+m+4ypgtogUNtJd+LL2CePHnj1Z3HnnR3Ts+DI//LCJN95YQnZ2rtdhGROW/JUorgLaqep+EakFzAReDU1YZcDaJ0whVJWpU5dx552z2LRpH9HRwvDhnXj44bOsmsmYIvhLFFmquh9AVbeJSGRdF7jTEoX5s717s7jiiql89NEqAE49tR5paX1p3/4YjyMzJrz5SxTH+YyVLUBT37GzVbVfUCM7WlaiMAVUrhxHVlYu1arF88QT53DzzScTFWVdgBtTHH+J4pIC0+ODGUiZys12GrMRqNnS62iMh+bN+526dSvTvHkyIsKrr15AQkIMdepU9jo0YyKGvzGzPwtlIGUq4zfIy4Fqx0FsktfRGA9s336Ae+6ZzT//uYju3Zswe/ZARIRGjap7HZoxEad8dpxv1U4VVl6e8tprixgxYjY7dx4kLi6aM89sSG6uEhNj1UzGlEZQG6hFpJeIrBCRVSJyr5/1LhERFZGy6T/KEkWFtHTpVrp1e41Bg6axc+dBundvws8/38ro0d2IiYmsazGMCScBlyhEJF5Vs0qwfjQwATgXSAcWiMg0336j3PWqAHcC3wW672JZoqhwdu/OpFOnV9i37xC1a1di7NgeXHllWxuv2pgyUOzPLBHpKCI/A7+50yeKSCBdeHTEGbtijaoeAt4CLixkvUeBJ4HMwMMuhiWKCkNVAahWLYGRIzszePDJLF9+G1dd1c6ShDFlJJDy+PNAX2AHgKouBs4KYLt6wAaf6XQKjLUtIicBDVR1hr8dicjNIrJQRBZu27bN/1HzctwBi7ArnsqxjRv3cOml7zBlypLD80aNOpNJk/pSo4b18mpMWQokUUSp6u8F5h11XwfuDXxjgbuLW1dVX1TVVFVNrVWrlv+VM9ZA7iGo0hDibAzj8iYnJ49x4+bTsuUE/vvfXxk9ei65uXkAVoIwJkgCaaPYICIdAXXbHW4HVgaw3Uaggc90fXdevipAG2Cu+wE/BpgmIheo6sJAgi/UjqXOY8oJpd6FCU8LFmxk8OAZ/PjjJgAuuqglzz/fi+hoa6g2JpgCSRS34lQ/NQS2AJ+684qzAGguIk1wEkR/4Mr8haq6G0jJnxaRucD/HVWSAOsMsBzav/8QI0d+ysSJC1CFhg2r8cIL53HBBdZ9vDGhUGyiUNWtOF/yJaKqOSIyFPgYiAZeVdWlIvIIsFBVp5U42kBYQ3a5ExMTxaefriEqShg+/DRGj+5KpUqRMciiMeVBsYlCRF4CtOB8Vb25uG1VdSZOr7O+8x4sYt1uxe0vIJYoyoXVq3dSvXoCyclJxMfH8MYbF5OQEEPbtnW8Ds2YCieQyt1Pgc/cv6+B2kDA91OEVF4u7FruPE9u5W0splSysnJ47LF5tGkziZEjPz08/5RT6lmSMMYjgVQ9ve07LSJvAF8FLaKjsWcd5GRC5XoQX83raEwJzZ27jltvncHy5dsB5wqn3Nw8a6w2xmOl6eupCRCeP+2s2ikibd26nxEjZvP664sBaNEimUmT+nDWWU08jswYA4G1UeziSBtFFLATKLLfJk9tdy+NTbZLYyPF9u0HaNVqAjt3HiQ+PppRo87knns6Ex9fPvurNCYS+f00inODw4kcuf8hT/P7TAhHNqpdxElJSeLCC1uQnr6HiRP70KxZTa9DMsYU4DdRqKqKyExVbROqgI6KVT2Fvf37D/HII1/Qp8/xdOnSCICJE/sQHx9td1YbE6YCaSVcJCIdgh7J0dI82PGr87ymXfEUjj78cAWtW09kzJhvGDJkBnl5TuE0ISHGkoQxYazIEoWIxKhqDtABp4vw1cB+nPGzVVVPClGMgdmzHnIOQKVjINGqL8LJhg27ufPOWbz3nnPpcocOxzB5cl8br9qYCOGv6ul74CTgghDFcnSs2ins5OTk8fzz3/Hgg3PYvz+bypXjeOyxs7jtto42kJAxEcRfohAAVV0doliOjvXxFHb27MniH//4iv37s7nkklY891wv6tev6nVYxpgS8pcoaonI8KIWqurYIMRTetZrbFjIyMgkMTGG+PgYatZMZPLkvsTHR9Onz/Feh2aMKSV/5f9ooDJOd+CF/YUXq3rylKry5ps/06LFeMaM+frw/H79WlmSMCbC+StRbFLVR0IWydFQtaonD61cuYMhQ2bw2WdrAZg3bz2qalcyGVNOFNtGERH2pkP2PkisBUkpxa9vykRmZg5PPvkVf//7Vxw6lEvNmok89dS5XHdde0sSxpQj/hJF95BFcbTsjuyQ27x5H126/JPfftsJwHXXteepp84lJSXJ48iMMWWtyEShqjtDGchRsfaJkKtTpxINGlQjJiaKSZP60LVrY69DMsYESfnoec0SRdDl5SkvvfQDZ53VhOOPT0ZEePPNftSokUhcXLTX4Rljgqh83PVkvcYG1eLFm+nc+VUGD57BkCEzyO8Xsk6dypYkjKkAIr9EoWptFEGyb98hHnpoLs89N5/cXOXYY6sweHCq12EZY0Is8hPF/k2QtRsSakJSba+jKTfef385t9/+Eenpe4iKEm6/vSOPPXY2VavGex2aMSbEIj9R+LZP2CWZZWLjxj307z+VrKxcTj65LmlpfUlNPdbrsIwxHilficKUWnZ2LjExUYgI9epV5fHHzyYuLpohQ06xMauNqeAi/xvAEsVR++abDZx88otMmbLk8Ly77z6d228/1ZKEMaYcJQrruqPEdu48yC23fEjnzq/y889bmThxIeE80q0xxhuRX/WUscp5rGkdzwVKVZkyZQl33/0J27YdIDY2invu6cyoUWda1xvGmL+I7ESRlwsHtjrPK9X1NpYIsWXLPgYM+C9z5qwDoGvXRkya1IdWrWp5G5gxJmxFdqI4uB00FxKSITrO62giQvXqCWzatI+UlCSefvpcrrnmRCtFGGP8iuxEsX+T81jZShP+zJ69mpNOqktychLx8TG8++5l1K1bmeRk68DPGFO8yG7M3r/ZeUw6xts4wtSmTXsZMOC/9OgxhZEjPz08v02b2pYkjDEBsxJFOZSbm8fkyT9w332fsWdPFomJMbRokWyDCRljSqV8JApryD7sxx83MXjwdBYs+AOAPn2aM358bxo3ru5xZMaYSBXhicKteqpkVU8A69Zl0LHjS+TmKvXqVeH558/j4otbWinCGHNUgpooRKQXMA6IBl5W1ScKLB8O3AjkANuAG1T194APYCWKP2ncuDrXX9+eKlXiefjhblSpYh34GWOOXtAas0UkGpgAnAe0BgaISMHbp38CUlW1HTAVGFOig+yr2Ili3boMzj//P3zxxbrD81588XzGju1pScIYU2aCWaLoCKxS1TUAIvIWcCGwLH8FVZ3js/584OoSHeFAxax6ys7OZezYb3n44S84eDCH7dsP8O23gwCsmskYU+aCmSjqARt8ptOBU/2sPwj4qLAFInIzcDNAw4YNnZmqFbJE8dVX6xk8eDpLl24DoH//Nowd28PjqIwx5VlYNGaLyNVAKtC1sOWq+iLwIkBqaqrTa92hvZBzAGKSIK5KqEL1zK5dBxkxYjavvPITAE2b1mDixD706NHU48iMMeVdMBPFRqCBz3R9d96fiMg5wCigq6pmBbx33yueKkB1S16e8sEHK4iNjeLee8/gvvvOIDEx1uuwjDEVQDATxQKguYg0wUkQ/YErfVcQkQ7AZKCXqm4t0d4rwBVPy5dvp0mT6sTHx5CcnMS//92Phg2r0bJlitehGWMqkKBd9aSqOcBQ4GPgV+AdVV0qIo+IyAXuak8BlYF3RWSRiEwL+AD5JYpyeFf2gQPZjBr1Ge3aTWLMmK8Pz+/Ro6klCWNMyAW1jUJVZwIzC8x70Of5OaXeeX6Jopz18zRr1iqGDJnB2rUZAGzffsDjiIwxFV1YNGaXSjnr5+mPP/YybNgs3n3XuXq4bdvapKX15fTTGxSzpTHGBFcEJ4r8xuzITxQrV+4gNfVF9u49RFJSLA891JVhwzoRGxvtdWjGGBPJiSK/MTvyq56aN6/JKafUo1KlWF544TwaNbIO/Iwx4aMcJIrIK1Hs2ZPFgw/OYciQUzj++GREhGnT+lOpko3SZ4wJPxGcKCKv6klVmTp1GXfeOYtNm/axfPl2Zs1yei2xJGGMCVeRmShyDznjZUsUJEbG5aJr1uxi6NCZfPTRKgA6darPk0+W/qIvY4wJlchMFPu3OI9JdSAqvBt8Dx3K5emnv+HRR+eRmZlD9eoJPPFEd2666WSiosr/HeXGmMgXmYkignqN3bBhN4888gVZWblcdVVbnnmmB3XqVPY6LGOMCVhkJoow7zV2166DVK+egIjQtGlNxo3rRbNmNene/TivQzPGmBILWhceQRWmVzzl5SmvvvoTzZq9wJQpSw7Pv+WWVEsSxpiIFaGJIvyqnpYu3Uq3bq8xaNA0du48eLjR2hhjIl1kVj2FUYniwIFsHn30C55++ltycvKoXbsSzz7bkwED2ngdmjHGlInIThQe9/O0cuUOevacwrp1GYjA4MEn8/e/d6dGjURP4zLGmLIUoYnCrXryuOfYRo2qkZAQw4kn1iEtrS+dOtX3NB5jjAmGCE0U3pQocnLySEtbyIABbUhOTiI+PoZZs66iXr2qxMREZnOPMcYUJ0ITRehLFN9/v5HBg6fz00+bWbRoMy+/7Iy9ZB34GWPKu8hLFHk5kJcN8dUgNvhtAbt3ZzJq1OdMnLgAVWjYsBoXXtgi6Mc1xphwEYGJItt5DPIVT6rK228v5a67Pmbz5n3ExEQxfHgnHnywq3XgZ4ypUCxRFGHx4i0MGPBfAE4/vQFpaX1o27ZOUI9pjDHhKIITRdm3T+Tm5hEd7TRKt29/DHfd1YnWrWtxww0drAM/Y0yFFXmX6uQGp0QxZ85a2rSZxLx5vx+eN3ZsT2688SRLEsaYCi3yEkUZVz1t3bqfa699n7PPfp3ly7czduy3ZbJfY4wpLyps1VNenvLKKz8ycuSn7NqVSXx8NPff34URI04vgyCNMab8iOBEUfoSxdq1u7j66vf45psNAPTo0ZQJE3rTrFnNsojQGGPKlchLFPltFEdxV3bVqvGsXLmDY46pzHPP9eTyy09AxNohjDGmMJGXKPJLFCW8K/vjj1fRrVtj4uNjSE5OYtq0/rRuXYtq1RKCEKQxxpQfkdeYrXkQHQ8JNQJafcOG3Vx88dv06vVvnnrqm8PzTzutgSUJY4wJQOSVKMBpyC6mqignJ4/nn/+OBx+cw/792VSuHEfNmtb9tzHGlFTkJgo/5s9PZ/Dg6SxevAWASy5pxbhxvahXr2ooojPGmHIlQhNF0Q3Z332Xzumnv4IqNG5cnfHjz6NPn+NDGJwxxpQv5S5RdOxYj549m9GhwzHcf38XkpJiQxiYMcaUP5HXmA1/qnr67bcd9O37JitX7gBARJgx40r+/vfuliSMMaYMRGyJIisrhyee+Ip//OMrsrJySUiIYerUywGsbyZjjClDQS1RiEgvEVkhIqtE5N5ClseLyNvu8u9EpHEg+120SmjXLo2HHvqCrKxcrr++PWlpfcs6fGOMMQSxRCEi0cAE4FwgHVggItNUdZnPaoOAXaraTET6A08CVxS37xvv/J6V6fVo1SqFtLS+dOnSKBgvwRhjDMEtUXQEVqnqGlU9BLwFXFhgnQuBf7nPpwLdJYC+NHZl1eDvfz+bRYsGW5IwxpggE1UNzo5FLgV6qeqN7vRA4FRVHeqzzi/uOunu9Gp3ne0F9nUzcLM72Qb4JShBR54UYHuxa1UMdi6OsHNxhJ2LI1qoapXSbBgRjdmq+iLwIoCILFTVVI9DCgt2Lo6wc3GEnYsj7FwcISILS7ttMKueNgINfKbru/MKXUdEYoBqwI4gxmSMMaaEgpkoFgDNRaSJiMQB/YFpBdaZBlzrPr8U+FyDVRdmjDGmVIJW9aSqOSIyFPgYiAZeVdWlIvIIsFBVpwGvAG+IyCpgJ04yKc6LwYo5Atm5OMLOxRF2Lo6wc3FEqc9F0BqzjTHGlA+R2YWHMcaYkLFEYYwxxq+wTRTB6v4jEgVwLoaLyDIRWSIin4lIub0Lsbhz4bPeJSKiIlJuL40M5FyIyOXue2OpiLwZ6hhDJYDPSEMRmSMiP7mfk95exBlsIvKqiGx171ErbLmIyPPueVoiIicFtGNVDbs/nMbv1cBxQBywGGhdYJ0hQJr7vD/wttdxe3guzgKS3Oe3VuRz4a5XBZgHzAdSvY7bw/dFc+AnoIY7XdvruD08Fy8Ct7rPWwPrvI47SOeiC3AS8EsRy3sDHwECdAK+C2S/4VqiCFr3HxGo2HOhqnNU9YA7OR/nnpXyKJD3BcCjOP2GZYYyuBAL5FzcBExQ1V0Aqro1xDGGSiDnQoH8IS6rAX+EML6QUdV5OFeQFuVC4HV1zAeqi0jRA/y4wjVR1AM2+Eynu/MKXUdVc4DdQHJIogutQM6Fr0E4vxjKo2LPhVuUbqCqM0IZmAcCeV8cDxwvIl+LyHwR6RWy6EIrkHPxEHC1iKQDM4HbQxNa2Cnp9wkQIV14mMCIyNVAKtDV61i8ICJRwFjgOo9DCRcxONVP3XBKmfNEpK2qZngalTcGAK+p6jMichrO/VttVDXP68AiQbiWKKz7jyMCOReIyDnAKOACVc0KUWyhVty5qILTaeRcEVmHUwc7rZw2aAfyvkgHpqlqtqquBVbiJI7yJpBzMQh4B0BVvwUScDoMrGgC+j4pKFwThXX/cUSx50JEOgCTcZJEea2HhmLOharuVtUUVW2sqo1x2msuUNVSd4YWxgL5jLyPU5pARFJwqqLWhDLIEAnkXKwHugOISCucRLEtpFGGh2nANe7VT52A3aq6qbiNwrLqSYPX/UfECfBcPAVUBt512/PXq+oFngUdJAGeiwohwHPxMdBDRJYBucAIVS13pe4Az8XdwEsichdOw/Z15fGHpYj8B+fHQYrbHjMaiAVQ1TSc9pnewCrgAHB9QPsth+fKGGNMGQrXqidjjDFhwhKFMcYYvyxRGGOM8csShTHGGL8sURhjjPHLEoUJOyKSKyKLfP4a+1m3cVE9ZZbwmHPd3kcXu11etCjFPgaLyDXu8+tE5FifZS+LSOsyjnOBiLQPYJthIpJ0tMc2FZclChOODqpqe5+/dSE67lWqeiJOZ5NPlXRjVU1T1dfdyeuAY32W3aiqy8okyiNxTiSwOIcBlihMqVmiMBHBLTl8KSI/un+nF7LOCSLyvVsKWSIizd35V/vMnywi0cUcbh7QzN22uzuGwc9uX//x7vwn5MgYIE+78x4Skf8TkUtx+tz6t3vMRLckkOqWOg5/ubslj/GljPNbfDp0E5FJIrJQnLEnHnbn3YGTsOaIyBx3Xg8R+dY9j++KSOVijmMqOEsUJhwl+lQ7vefO2wqcq6onAVcAzxey3WBgnKq2x/miTne7a7gC6OzOzwWuKub45wM/i0gC8Bpwhaq2xenJ4FYRSQYuBk5Q1XbAY74bq+pUYCHOL//2qnrQZ/F/3W3zXQG8Vco4e+F005FvlKqmAu2AriLSTlWfx+lS+yxVPcvtyuN+4Bz3XC4EhhdzHFPBhWUXHqbCO+h+WfqKBca7dfK5OP0WFfQtMEpE6gP/U9XfRKQ7cDKwwO3eJBEn6RTm3yJyEFiH0w11C2Ctqq50l/8LuA0YjzPWxSsiMh2YHugLU9VtIrLG7WfnN6Al8LW735LEGYfTbYvvebpcRG7G+VzXxRmgZ0mBbTu58792jxOHc96MKZIlChMp7gK2ACfilIT/MiiRqr4pIt8BfYCZInILzkhe/1LV+wI4xlW+HQiKSM3CVnL7FuqI08ncpcBQ4OwSvJa3gMuB5cB7qqrifGsHHCfwA077xAtAPxFpAvwfcIqq7hKR13A6vitIgNmqOqAE8ZoKzqqeTKSoBmxyxw8YiNP525+IyHHAGre65QOcKpjPgEtFpLa7Tk0JfEzxFUBjEWnmTg8EvnDr9Kup6kycBHZiIdvuxen2vDDv4Yw0NgAnaVDSON0O7R4AOolIS5zR2/YDu0WkDnBeEbHMBzrnvyYRqSQihZXOjDnMEoWJFBOBa0VkMU51zf5C1rkc+EVEFuGMS/G6e6XR/cAnIrIEmI1TLVMsVc3E6V3zXRH5GcgD0nC+dKe7+/uKwuv4XwPS8huzC+x3F/Ar0EhVv3fnlThOt+3jGZxeYRfjjI+9HHgTpzor34vALBGZo6rbcK7I+o97nG9xzqcxRbLeY40xxvhlJQpjjDF+WaIwxhjjlyUKY4wxflmiMMYY45clCmOMMX5ZojDGGOOXJQpjjDF+/T/i7g2Hxo9dqwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "# ROC Curve 를 출력한다\n",
    "import matplotlib.pyplot as plt\n",
    "lw = 2   # BaseLine을 그린다\n",
    "plt.plot([0, 1], [0, 1], color='navy',  lw=lw, linestyle='--')\n",
    "plt.plot(false_pos_rate, true_pos_rate, color = 'darkorange', lw = lw)\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver Operating Characteristic')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9629610085418291"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "roc_auc_score(Y_test, pos_prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br/>\n",
    "\n",
    "## **6 Sklearn 을 활용한 모델의 튜닟 및 교차검증**\n",
    "1. 모델이 실질적으로 잘 작동하는지 **K-fold 검정을** 적용한다\n",
    "1. **AUC 값의** 측정 : **ROC 커브의** 밑면적을 구한 값으로 **1에 가까울수록** 성능이 좋다.[참고](http://newsight.tistory.com/53)\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/1/1c/K-fold_cross_validation_EN.jpg\" align=\"left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 전체 10개의 폴드 생성기로 초기화 후 파라미터 분석을 진행합니다\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "k = 10\n",
    "k_fold = StratifiedKFold(n_splits=k)\n",
    "\n",
    "# 연산을 위해 Numpy 객체로 변환한다\n",
    "cleaned_emails_np = np.array(cleaned_emails)\n",
    "labels_np         = np.array(labels)\n",
    "\n",
    "# 10 폴드 생성기 학습을 위한 파라미터를 정의합니다\n",
    "max_features_option     = [2000, 4000, 8000]   # 가장 많이 사용되는 N개 단어를 선택\n",
    "smoothing_factor_option = [0.5, 1.0, 1.5, 2.0] # Smoothing Parameter : 초기값\n",
    "fit_prior_option        = [True, False]        # 사전 확률을 사용할지 여부"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 24.8 s, sys: 1.16 s, total: 26 s\n",
      "Wall time: 26 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "auc_record = {} # k_fold 분리된 객체를 활용하여 개별 환경에서 AUC를 측정\n",
    "for train_indices, test_indices in k_fold.split(cleaned_emails, labels):\n",
    "    X_train, X_test = cleaned_emails_np[train_indices], cleaned_emails_np[test_indices]\n",
    "    Y_train, Y_test = labels_np[train_indices], labels_np[test_indices]\n",
    "\n",
    "    # max_features_option 환경값을 바꿔가면서 AUC 테스트\n",
    "    for max_features in max_features_option:        \n",
    "        if max_features not in auc_record:\n",
    "            auc_record[max_features] = {}\n",
    "        cv = CountVectorizer(stop_words=\"english\", max_features=max_features)\n",
    "        term_docs_train = cv.fit_transform(X_train)\n",
    "        term_docs_test  = cv.transform(X_test)\n",
    "        \n",
    "        # smoothing_factor_option 초기값을 바꾸며 AUC 테스트\n",
    "        for smoothing_factor in smoothing_factor_option:\n",
    "            if smoothing_factor not in auc_record[max_features]:\n",
    "                auc_record[max_features][smoothing_factor] = {}\n",
    "            \n",
    "            # fit_prior_option : 사전확률을 바꾸며 AUC 테스트\n",
    "            for fit_prior in fit_prior_option:\n",
    "                clf = MultinomialNB(alpha=smoothing_factor, fit_prior=fit_prior)\n",
    "                clf.fit(term_docs_train, Y_train)\n",
    "                prediction_prob = clf.predict_proba(term_docs_test)\n",
    "                pos_prob        = prediction_prob[:, 1]\n",
    "                auc             = roc_auc_score(Y_test, pos_prob)\n",
    "                auc_record[max_features][smoothing_factor][fit_prior] \\\n",
    "                    = auc + auc_record[max_features][smoothing_factor].get(fit_prior, 0.0)                \n",
    "\n",
    "# 위에서 계산한 결과를 출력합니다\n",
    "auc_result = []\n",
    "for max_features, max_feature_record in auc_record.items():\n",
    "    for smoothing, smoothing_record in max_feature_record.items():\n",
    "        for fit_prior, auc in smoothing_record.items():\n",
    "            auc_result.append([max_features, smoothing, fit_prior, auc/k])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max features</th>\n",
       "      <th>smoothing</th>\n",
       "      <th>fit prior</th>\n",
       "      <th>auc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8000</td>\n",
       "      <td>0.5</td>\n",
       "      <td>True</td>\n",
       "      <td>0.988589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8000</td>\n",
       "      <td>0.5</td>\n",
       "      <td>False</td>\n",
       "      <td>0.988520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>True</td>\n",
       "      <td>0.987575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "      <td>0.987436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8000</td>\n",
       "      <td>1.5</td>\n",
       "      <td>True</td>\n",
       "      <td>0.987039</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   max features  smoothing  fit prior       auc\n",
       "0          8000        0.5       True  0.988589\n",
       "1          8000        0.5      False  0.988520\n",
       "2          8000        1.0       True  0.987575\n",
       "3          8000        1.0      False  0.987436\n",
       "4          8000        1.5       True  0.987039"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "auc_result = pd.DataFrame(auc_result)\n",
    "auc_result.columns = ['max features', 'smoothing', 'fit prior', 'auc']\n",
    "auc_result = auc_result.sort_values('auc', ascending=False).reset_index(drop=True)\n",
    "auc_result.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}