seq2seq-roman.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# demonstration of sequence-to-sequence learning with attention\n",
    "\n",
    "## background: sequence-to-sequence models\n",
    "\n",
    "recurrent sequence-to-sequence models are a type of \"many to many\" ([karpathy 2015](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)) recurrent network in an encoder-decoder configuration. first, a recurrent network 'encodes' the input sequence, and then a recurrent decoder uses information from the encoder to generate an output sequence. this differs from recurrent models for POS tagging, entity recognition, etc. because it allows for input and output sequences of arbitrary and differing lengths. this is useful for tasks such as translation, where a sentence of *x* tokens (e.g. words, morphemes, characters) in the input language may have a translation of *y* tokens in the output language, where *x* may not equal *y*.\n",
    "\n",
    "in the basic sequence-to-sequence model, all information from the encoder is passed to the decoder network through state transfer, where the final recurrent encoder state(s) are used to initialize the decoder state. we use the final hidden state *h* and cell state *c* from each encoder LSTM layer as the initial state for the decoder LSTM, but there are many alternatives and variations to this; the [wanasit article](https://wanasit.github.io/attention-based-sequence-to-sequence-in-keras.html) upon which webase our attention implementation uses the final encoder *output state* to initialize both the cell and hidden state of the decoder in their one-layer LSTM model, and for an alternative approach, the [keras blog post](https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html) demonstrates a seq2seq model where inversely the final encoder state serves as the *input* *x* to the decoder at each timestep (the first model, before the main demonstration). however both these approaches rely on a single fixed-size representation of the encoder information.\n",
    "\n",
    "a more effective approach may be to allow the network to \"look back\" at the encoder information at each timestep, but unlike the keras blog's basic implementation, allow the network to \"focus\" on different words at each timestep. this is implemented with *attention*.\n",
    "\n",
    "## background: attention\n",
    "\n",
    "attention is borrowed from work on computer vision, which was itself inspired by analogies to human visual processes. *Attention* instead considers a *weighted sum* over each timestep *t* (= token) of the input for each decoder timestep *s*, allowing the network to \"focus on\" different parts of the input when generating each token of the output. the analogy would be manually translating a sentence: in basic seq2seq, you only get to read the sentence one time, and then you must translate the whole sentence at once. with attention, you are allowed to look back at the source sentence before writing each word of the translation, and naturally you would focus on the word or words correlating with the part of the sentence you are writing next, ignoring words irrelevant to the current context. this 'focus' is captured in the attention matrix, which is multiplied by each output vector of the timestep, with higher weights corresponding to higher 'focus'. \n",
    "\n",
    "while the attention seen in _Bahdanau, Cho and Bengio 2015_ \"Neural Machine Translation by Jointly Learning to Align and Translate\" conducts this weighted sum on the *input* to the decoder RNN at each timestep, due to the fixed-graph restriction of `tensorflow` and the layer-wise approach taken in `keras`, the \"global attention\" from _Luong, Pham, Manning 2015_ \"Effective Approaches to Attention-based Neural Machine Translation\", particularly the simple \"dot\" attention, will allow for a simpler implementation. this implementation combines the 'basic' seq2seq decoder output state with  \n",
    "\n",
    "## task: transliteration from Korean to English\n",
    "\n",
    "we will build a toy sequence-to-sequence model that will transliterate Korean words in Hangul to the [McCune-Reischauer romanization](https://www.library.illinois.edu/ias/koreancollection/koreanromanizationtable/). on a linguistic/cultural note, this romanization system is no longer favored, with Korean linguistics using the [Yale format](https://en.wikipedia.org/wiki/Yale_romanization_of_Korean) and the South Korean government using its own [revised romanization](https://en.wikipedia.org/wiki/Revised_Romanization_of_Korean) for things such as street signs, documents, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## data processing\n",
    "\n",
    "here we will reead in the data, split the Korean into a sequence of graphemes, and split the English into a sequence of letters (\"tokenization\").\n",
    "\n",
    "data is from: https://github.com/digitalprk/mcr_romanization\n",
    "\n",
    "grapheme splitting code is from: https://github.com/Kcrong/separate-korean (thanks kcrong)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "import re\n",
    "from separator import Separator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>화랑도</td>\n",
       "      <td>hwarangdo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>해심</td>\n",
       "      <td>haesim</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>쨄</td>\n",
       "      <td>tchael</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>쇡</td>\n",
       "      <td>soek</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>언제쯤</td>\n",
       "      <td>ŏnjetchŭm</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0          1\n",
       "0  화랑도  hwarangdo\n",
       "1   해심     haesim\n",
       "2    쨄     tchael\n",
       "3    쇡       soek\n",
       "4  언제쯤  ŏnjetchŭm"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read the csv using pandas\n",
    "# separator is a tab (\\t) and there are no column header rows\n",
    "csvdata = pd.read_csv('data/wordlist.csv', sep='\\t', header=None)\n",
    "csvdata.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get each column as a python list\n",
    "kor_words = csvdata[0].tolist()\n",
    "eng_words = csvdata[1].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ㅎ', 'ㅘ', 'ㄹ', 'ㅏ', 'ㅇ', 'ㄷ', 'ㅗ']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test the Separator\n",
    "Separator(kor_words[0]).sep_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# non-Korean characters will cause Separator to error\n",
    "# so we must remove all non-Korean characters, including spaces\n",
    "kor_words = [re.sub(r'[^가-힣ㄱ-ㅎ]', '', word) for word in kor_words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenize the Korean into lists of graphemes\n",
    "kor_toks = [Separator(word).sep_all for word in kor_words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenize the English into lists of characters\n",
    "# we lower-case and replace spaces with an underscore\n",
    "eng_toks = [list(str(sent).lower().replace(' ', '_')) for sent in eng_words]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## data indexing\n",
    "\n",
    "we will then truncate and/or \"pad\" the input and output to a fixed length of our choosing, create frequency-based character mappings, and convert the tokenized input to integer vectors, for input into the `keras` network.\n",
    "\n",
    "more advanced training methods will \"bucket\" the data by similar lengths before padding, and then construct a set of fixed-length networks with shared weights, but for this simple demonstration we can use the `keras` zero-padding and set all data to the same length, which we base on the mean length plus two standard deviations (assuming a normal distribution, we should capture ~97% of the sentences without truncating)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KOR: total unique tokens: 51\n",
      "ENG: total unique tokens: 70\n"
     ]
    }
   ],
   "source": [
    "# get the number of unique tokens in each language\n",
    "print('KOR: total unique tokens:', len(set([word for sent in kor_toks for word in sent])))\n",
    "print('ENG: total unique tokens:', len(set([word for sent in eng_toks for word in sent])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KOR: maximum string len : 26\n",
      "KOR: avg len + 2 stdevs : 10.99392885818844\n",
      "\n",
      "ENG: avg len + 2 stdevs : 13.126664572515036\n",
      "ENG: maximum string len : 29\n"
     ]
    }
   ],
   "source": [
    "# get the maximum input lengths for each language\n",
    "# assuming normal distribution, mean + 2 stds = 97.8% of lengths\n",
    "# add one for SOS tag\n",
    "import numpy as np\n",
    "print('KOR: maximum string len :', max([len(s)+1 for s in kor_toks]))\n",
    "print('KOR: avg len + 2 stdevs :', np.mean([len(s)+1 for s in kor_toks]) + 2 * np.std([len(s) for s in kor_toks]))\n",
    "print()\n",
    "print('ENG: avg len + 2 stdevs :', np.mean([len(s)+1 for s in eng_toks]) + 2 * np.std([len(s) for s in eng_toks]))\n",
    "print('ENG: maximum string len :', max([len(s)+1 for s in eng_toks]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_arrays(tokens, maxvocab=6000, maxlen=12, pad = '_PAD_', unk = '_UNK_', sos = 'Ⓑ', padfrom = 'end'):\n",
    "    \"\"\"integer-index and pad tokenized text\"\"\"\n",
    "    from collections import Counter\n",
    "    import numpy as np\n",
    "    \n",
    "    # get a single list of all words\n",
    "    words = [word for sent in tokens for word in sent]\n",
    "    # get the count of each word and sort by frequency (highest to lowest)\n",
    "    # this is just by convention\n",
    "    counts = Counter(words)\n",
    "    words = sorted(list(set(words)), key=counts.get, reverse=True)\n",
    "    # truncate to desired vocabulary and add PAD and UNK/OOV symbols\n",
    "    words = words[:maxvocab-3]\n",
    "    words.insert(0, sos)\n",
    "    words.insert(0, pad)\n",
    "    words.insert(-1, unk)\n",
    "    # create dictionaries\n",
    "    tok2idx = dict(zip(words, [i for i in range(len(words))]))\n",
    "    idx2tok = dict(zip([i for i in range(len(words))], words))\n",
    "    \n",
    "    # index each sentence\n",
    "    idxes = []\n",
    "    for tok_sent in tokens:\n",
    "        # pad and truncate\n",
    "        tok_sent = tok_sent[:maxlen-1]\n",
    "        tok_sent.insert(0, sos)\n",
    "        while len(tok_sent) < maxlen:\n",
    "            if padfrom == 'end':\n",
    "                tok_sent.append(pad)\n",
    "            else:\n",
    "                tok_sent.insert(0, pad)\n",
    "        # convert to indices and add\n",
    "        idxes.append([tok2idx.get(word, tok2idx[unk]) for word in tok_sent])\n",
    "    \n",
    "    # convert to numpy array and return\n",
    "    return np.array(idxes), tok2idx, idx2tok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# index and pad\n",
    "kor_idxs, kor2idx, idx2kor = make_arrays(kor_toks, maxvocab=100, maxlen=12, padfrom = 'start')\n",
    "eng_idxs, eng2idx, idx2eng = make_arrays(eng_toks, maxvocab=100, maxlen=16, padfrom = 'end')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create output targets from decoder inputs\n",
    "# this shifts the alignment my one (= predict *next* character) and adds extra dim\n",
    "eng_outs = np.expand_dims(np.hstack((eng_idxs[:,1:], np.zeros(shape=(eng_idxs.shape[0], 1)))), axis=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## sequence to sequence network\n",
    "\n",
    "this is based on:  \n",
    "_Sutskever, Vinyals, Le 2014_ \"[Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215)\"  \n",
    "_Luong, Pham, Manning 2015_ \"[Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/pdf/1508.04025.pdf)\"\n",
    "\n",
    "attention code is based on https://wanasit.github.io/attention-based-sequence-to-sequence-in-keras.html with modifications to bring it in line with _Sutskever et al._\n",
    "\n",
    "### notes on hyperparameters\n",
    "\n",
    "while the original word2vec papers found that word vectors of size 300 were efficient, and seq2seq translation models use embedding and RNN vector sizes of 600, 1024, or more, for this simple toy task, a size of 100 will be sufficient. as to what this embedding size is, it is the number of latent features that we will allow the network to represent each character with. for the recurrent network, this size will represent the size of the 'memory' or 'state' that persists through each timestep. the embedding size and recurrent size may differ.\n",
    "\n",
    "in tasks where the input and output sequences are from the same space (i.e. the same language, for example in a summarization task), we can use the same embedding layer for both the encoder and decoder. however, because we are drawing from two different source 'languages', we will use separate vector spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, Dropout, Embedding, LSTM \n",
    "from keras.layers import Activation, dot, TimeDistributed\n",
    "from keras.layers import concatenate, Dense, Bidirectional\n",
    "from keras.models import model_from_json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyperparameters\n",
    "KOR_VOCAB  = len(kor2idx)      # how many unique words in the input language\n",
    "KOR_EMBED  = 100               # low long are the character vectors in our input embedding space\n",
    "MAX_IN_LEN = kor_idxs.shape[1] # how long is the sentence vector\n",
    "\n",
    "ENG_VOCAB    = len(eng2idx)    # how many unique words in the output language\n",
    "ENG_EMBED    = 100             # low long are the character vectors in our input embedding space\n",
    "MAX_OUT_LEN  = eng_idxs.shape[1]\n",
    "\n",
    "# rnn size\n",
    "HIDDEN_SIZE = 100              # how big is the recurrent cell\n",
    "DROP_RATE   = 0.33             # what is our dropout frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## encoder\n",
    "\n",
    "we will use a two-layer bidirectional LSTM for encoding. each bi-directional LSTM layer consists of two LSTM networks, a *forward* LSTM and *backward* LSTM. as their names imply, the forward LSTM reads the input sequence in from beginning to end, while the backward LSTM reads the reverse sequence, from the end to the beginning. at each timestep, both LSTMs produce a source output state vector *h_s*, $h_{fs}$ and $h_{bs}$. the first layer's output state vector sequence is then the input to the second layer LSTM. \n",
    "\n",
    "to ensure that the encoder output vectors are the same size as the non-bidirectional decoder (which will be important for the attention mechanism), we will divide the `HIDDEN_SIZE` by two for the forward and backward networks, as the default joining method of the `Bidirectional` wrapper is 'end-to-end' concatenation by 'stacking' the forward and backward output vectors at each timestep. we will use `return_sequences=True` so that we can get the intermediate output state at each timestep, and `return_state=True` to get the final recurrent cell states. we then concatenate the forward and backward `h` hidden and `c` cell states _for each layer_ to create the initial states for our two-layer decoder.\n",
    "\n",
    "in summary, this encoder will take in a vector $x$ and return a sequence of output state vectors $\\bar{h}_s$, as well as the $h$ and $c$ state vectors for each layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bidirectional encoder\n",
    "encoder_input = Input(shape=(MAX_IN_LEN,), name='encoder_input')\n",
    "\n",
    "# korean embedding layer and dropout\n",
    "encoder_embed = Embedding(KOR_VOCAB, KOR_EMBED, mask_zero=True, name='encoder_embed')(encoder_input)\n",
    "encoder_embed = Dropout(DROP_RATE)(encoder_embed)\n",
    "\n",
    "# two-layer bidirectional LSTM with final states\n",
    "# divide the HIDDEN_SIZE by two because this is TWO LSTMS stacked\n",
    "encoder_hout1, fwd_h1, fwd_c1, bck_h1, bck_c1 = Bidirectional(LSTM(int(HIDDEN_SIZE/2), return_sequences=True, return_state=True), name='encoder_lstm1')(encoder_embed)\n",
    "encoder_hout2, fwd_h2, fwd_c2, bck_h2, bck_c2 = Bidirectional(LSTM(int(HIDDEN_SIZE/2), return_sequences=True, return_state=True), name='encoder_lstm2')(encoder_hout1)\n",
    "\n",
    "# state concatenation (h, c states for layers 1 and 2)\n",
    "state_h1 = concatenate([fwd_h1, bck_h1])\n",
    "state_c1 = concatenate([fwd_c1, bck_c1])\n",
    "state_h2 = concatenate([fwd_h2, bck_h2])\n",
    "state_c2 = concatenate([fwd_c2, bck_c2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## decoder\n",
    "\n",
    "the decoder can be viewed as a conditional language model, \"conditioned\" in the sense that it is learning the most likely output *given* an input sentence. this conditioned information is encoded by the encoder LSTM states (concatenated from the forward and backward LSTMs from each layer), which are set as the initial decoder states using the `initial_state` parameter.\n",
    "\n",
    "this network uses \"[teacher forcing](https://machinelearningmastery.com/teacher-forcing-for-recurrent-neural-networks/)\", meaning that the _desired_ token, not the actual previous generated token, is input at each timestep, during training. this has been shown to cause instability, and in a real network, alternatives prove to be better. but it works well on this toy example, and is simple to understand. so our `y_input` data and `y_output` data are vectorized representations of the same data, but offset by one, so that the `time_0` input is aligned to make the decoder *predict* the `time_1` output:\n",
    "\n",
    "```\n",
    "y_input : <start> hello   world     i     am      Derek   <end>\n",
    "\n",
    "          |       |       |       |     |       |        |\n",
    "\n",
    "y_output: hello   world     i       am    Derek   <end>   <pad>\n",
    "```\n",
    "\n",
    "so `<start>` will (hopefully) cause the network to predict `hello`, and then `hello` will cause the network to predict `world`, etc. etc. \n",
    "\n",
    "we will denote the decoder output state vectors as $h_t$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# decoder\n",
    "decoder_input = Input(shape=(MAX_OUT_LEN,), name='decoder_input')\n",
    "\n",
    "# english embedding layer and dropout\n",
    "decoder_embed = Embedding(ENG_VOCAB, ENG_EMBED, mask_zero=True, name='decoder_embed')(decoder_input)\n",
    "decoder_embed = Dropout(DROP_RATE)(decoder_embed)\n",
    "\n",
    "# two-layer LSTM initialized with encoder states\n",
    "decoder_hout1 = LSTM(HIDDEN_SIZE, return_sequences=True, name='decoder_lstm1')(decoder_embed, initial_state=[state_h1, state_c1])\n",
    "decoder_hout2 = LSTM(HIDDEN_SIZE, return_sequences=True, name='decoder_lstm2')(decoder_hout1, initial_state=[state_h2, state_c2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## *Luong et al.* attention\n",
    "\n",
    "The Luong attention works in the following manner:\n",
    "\n",
    "first, a *score* function measures the \"degree of alignment\" between each encoder output vector $\\bar{h}_s$ (s = \"source\") and each decoder output vector $h_t$ (t = \"timestep\"), constructing a matrix that is of size `output_length x input_length`. \n",
    "\n",
    "Luong *et al.* propose three score functions:\n",
    "\n",
    "$score( h_t, \\bar{h}_s) =  x =\\begin{cases}h_t^\\top \\bar{h}_s \\quad \\quad \\quad \\quad \\quad \\quad (dot)\\\\h_t^\\top W_a\\bar{h}_s \\quad \\quad \\quad \\quad \\quad (general)\\\\v_a^\\top tanh(W_a[h_t;\\bar{h}_s]) \\quad (concat)\\end{cases} $\n",
    "\n",
    "the *dot* method simply uses the dot product between the input and output vectors, which is similar to the *cosine similarity* metric used in `word2vec` papers to measure the angular distance between word vectors (the difference being that dot product factors in the magnitude as well). so we can think of this measurement as simply measuring the \"similarity\" of each input vector $\\bar{h}_s$ the the output vector $h_t$, with more similar tokens having a higher dot product. in translation, this is measuring the similarity between each word in the source sentence against the target word at time $t$; here we are measuring the similarity between a Korean grapheme and an English letter; so for example we would hope that `dot(ㅌ, t)` is high, and `dot(ㅝ, t)` it low.\n",
    "\n",
    "as for the other score functions, the *general* model first passes through a linearly-activated dense network with learned weights, and the *concat* model concatenates the input and output vectors before passing the concatenated vector through a `tanh`-activated dense network, and then multiplying by another weight vector *v*. we will try the *dot* method here, using `keras.layers.dot`. we must specify the `axes` as `[2, 2]` so that we do the dot product over the vectors $\\bar{h}_s$ and $h_t$ for *each* input and output timestep *s*, *t*.\n",
    "\n",
    "if we look at the `model.summary()` output, we see that this results in a tensor `attn_dotprod (Dot)` of size `(None, 16, 12)`. this is the unnormalized 'heat map' of attentions between the inputs and outputs, where each of the 16 rows in this matrix is a 12-length vector that represents the `score` or \"similarity\" between the *output* token of that *row index* and each of the 12 input tokens, which in turn represents where the network should \"focus its attention\" at that timestep.\n",
    "\n",
    "next, the attention matrix is found using equation [7] of the paper:\n",
    "\n",
    "$[7] \\quad a_t(s) = align(h_t, \\bar{h}_s) = \\frac{exp(score(h_t, \\bar{h}_s))}{ \\sum_s' exp(score(h_t, \\bar{h}_s'))} $\n",
    "\n",
    "which looks suspiciously like the softmax equation. By applying the softmax activation to the `score` tensor, each row now sums to `1.0`, so each value in that row demonstrates as a percentage how much we should focus on that input, at that output time.\n",
    "\n",
    "the `context vectors` $c_t$ for each output timestep *s* are found by multiplying each encoder output vector $\\bar{h}_s$ by the attention weight $a_t(s)$. this is a multiplication followed by a sum, so we can use the dot prodcut operation again. here use use `axes=[2, 1]`, meaning that we multiply each 12-length attention vector by the 12x100 encoder output matrix, for each of the 16 rows, resulting in a tensor of shape `(None, 16, 100)`. each 100-length column in this matrix represents the attention-weighted sum of the encoder (source) output vectors $\\bar{h}_s$ for that output timestep *t*.\n",
    "\n",
    "next, we can consider equations [5] and [6] in the paper:\n",
    "\n",
    "$[5] \\quad \\bar{h}_t = tanh(W_c[c_t;h_t])$\n",
    "\n",
    "so $\\bar{h}_t$, representing the intermediate prediction vector for timestep *t*, is a concatenation of the context vector $c_t$ (representing a weighted sum of the source sequence) and the decoder vector $h_t$, passed through a `tanh`-activated dense layer at each timestep. this is layer 'cont_dnstanh'.\n",
    "\n",
    "finally the output vector is a probability distribution over the output vocabulary, which is done using a softmax-activated dense layer:\n",
    "\n",
    "$[6] \\quad p(y_t|y_{<t}, x) = softmax(W_s\\bar{h}_t)$\n",
    "\n",
    "which is the final layer in out model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Luong global dot attention\n",
    "# score function from the Luong apper = dot\n",
    "score     = dot([decoder_hout2, encoder_hout2], axes=[2, 2], name='attn_dotprod')\n",
    "# turn score to \"attention dist.\" for weighted sum\n",
    "attention = Activation('softmax', name='attn_softmax')(score)\n",
    "\n",
    "# do the attention-weighted sum using dot product\n",
    "context   = dot([attention, encoder_hout2], axes=[2, 1], name='cont_dotprod')\n",
    "\n",
    "# 'stacked' the context vector with the decoder guess == 'attention vector'\n",
    "context   = concatenate([context, decoder_hout2], name='cont_concats')\n",
    "\n",
    "# activation\n",
    "context   = TimeDistributed(Dense(HIDDEN_SIZE*2, activation='tanh'), name='cont_dnstanh')(context)\n",
    "\n",
    "# guess which english letter\n",
    "output    = TimeDistributed(Dense(ENG_VOCAB, activation='softmax'))(context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# our model takes as input the encoder and decoder, and as target the shifted output we made\n",
    "model = Model([encoder_input, decoder_input], [output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## compilation and training\n",
    "\n",
    "we use the Adam optimizer, which is quite popular for this type of network. Because we are essentially doing a multi-class classification task ('classifying' the best output at each timestep where each letter is a 'class') and our final layer is using the `softmax` layer, we use the `categorical_crossentropy` loss, with `sparse` so that we do not need to convert our target vectors into one-hot vectors. we can then train using `fit()`.\n",
    "\n",
    "NB: the original blog uses `binary_crossentropy` loss and while the author achieves good results, the categorical cross-entropy is more fitting here, as our task is multi-class, as opposed to multi-label. the TL;DR is that the former will adjust each class independently. see [this article](https://gombru.github.io/2018/05/23/cross_entropy_loss/) for a brief discussion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compile the model with defined optimizer and loss function\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "encoder_input (InputLayer)      (None, 12)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "encoder_embed (Embedding)       (None, 12, 100)      5400        encoder_input[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "decoder_input (InputLayer)      (None, 16)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 12, 100)      0           encoder_embed[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "decoder_embed (Embedding)       (None, 16, 100)      7300        decoder_input[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "encoder_lstm1 (Bidirectional)   [(None, 12, 100), (N 60400       dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 16, 100)      0           decoder_embed[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 100)          0           encoder_lstm1[0][1]              \n",
      "                                                                 encoder_lstm1[0][3]              \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_2 (Concatenate)     (None, 100)          0           encoder_lstm1[0][2]              \n",
      "                                                                 encoder_lstm1[0][4]              \n",
      "__________________________________________________________________________________________________\n",
      "encoder_lstm2 (Bidirectional)   [(None, 12, 100), (N 60400       encoder_lstm1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "decoder_lstm1 (LSTM)            (None, 16, 100)      80400       dropout_2[0][0]                  \n",
      "                                                                 concatenate_1[0][0]              \n",
      "                                                                 concatenate_2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_3 (Concatenate)     (None, 100)          0           encoder_lstm2[0][1]              \n",
      "                                                                 encoder_lstm2[0][3]              \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_4 (Concatenate)     (None, 100)          0           encoder_lstm2[0][2]              \n",
      "                                                                 encoder_lstm2[0][4]              \n",
      "__________________________________________________________________________________________________\n",
      "decoder_lstm2 (LSTM)            (None, 16, 100)      80400       decoder_lstm1[0][0]              \n",
      "                                                                 concatenate_3[0][0]              \n",
      "                                                                 concatenate_4[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "attn_dotprod (Dot)              (None, 16, 12)       0           decoder_lstm2[0][0]              \n",
      "                                                                 encoder_lstm2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "attn_softmax (Activation)       (None, 16, 12)       0           attn_dotprod[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "cont_dotprod (Dot)              (None, 16, 100)      0           attn_softmax[0][0]               \n",
      "                                                                 encoder_lstm2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "cont_concats (Concatenate)      (None, 16, 200)      0           cont_dotprod[0][0]               \n",
      "                                                                 decoder_lstm2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "cont_dnstanh (TimeDistributed)  (None, 16, 200)      40200       cont_concats[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "time_distributed_1 (TimeDistrib (None, 16, 73)       14673       cont_dnstanh[0][0]               \n",
      "==================================================================================================\n",
      "Total params: 349,173\n",
      "Trainable params: 349,173\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## train and save"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# restrict GPU usage here, if need be\n",
    "import os\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "EPOCHS = 10\n",
    "\n",
    "history = model.fit([kor_idxs, eng_idxs], eng_outs,\n",
    "                    batch_size=64,\n",
    "                    epochs=EPOCHS,\n",
    "                    verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save\n",
    "\n",
    "# this is the model architecture (no weights)\n",
    "json_model = model.to_json()\n",
    "with open('kor2eng_{}_epochs.json'.format(EPOCHS), 'w') as outfile:\n",
    "    outfile.write(json_model)\n",
    "\n",
    "# this is model architecture (minus some params), and weights\n",
    "model.save('kor2eng_{}_epochs.h5'.format(EPOCHS))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.load_weights('kor2eng_{}_epochs.h5'.format(EPOCHS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## attention model\n",
    "\n",
    "by constructing a new model that outputs both a predicted output tensor `output` *and* the attention matrix tensor `attention`, we can construct a heatmap to visualize the attention. by using the same tensor names, we use the already trained model layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "heatmapmodel = Model([encoder_input, decoder_input], [output, attention])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## decode script\n",
    "\n",
    "this is just a function that accepts a string of Korean characters, preprocesses the string, generates a model prediction, converts the predictions back to English characters (using `np.argmax()` to find the most-activated class = predicted letter at each timestep), trims the 'padding' off of the attention matrix, and then renders the attention matrix as a heatmap.\n",
    "\n",
    "we use a 'greedy' encoding scheme, where at each timestep, we take the highest-predicted output (in line `output = output.argmax(axis=2)`) and append this to the `decoder_input` vector to generate the next character in the sequence. a \"better\" approach is \"[beam search](https://machinelearningmastery.com/beam-search-decoder-natural-language-processing/)\", which can be implemented here but is more computationally expensive, as at each timestep we must run the network `n` times to test each of the `n` best subsequences up to this timestep, where `n` is our \"beam width\".\n",
    "\n",
    "this code is extra ridiculous because non-English characters to not play well with `matplotlib` out of the box, and rendering Korean *graphemes* (as opposed to full characters) is also strange."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.font_manager as fm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate(kor_string, maxlen = 12, maxout = 16, sos = 'Ⓑ', unk = '_UNK_', pad = '_PAD_', padfrom = 'start'):\n",
    "    # tokenize, pad, and index\n",
    "    kor_tokens = [sos] + Separator(kor_string).sep_all\n",
    "    kor_tokens = kor_tokens[:maxlen]\n",
    "    \n",
    "    kor_toks = kor_tokens[:]\n",
    "    \n",
    "    while len(kor_tokens) < maxlen:\n",
    "        if padfrom == 'end':\n",
    "            kor_tokens.append(pad)\n",
    "        else:\n",
    "            kor_tokens.insert(0, pad)\n",
    "   \n",
    "    encoder_input = np.array([[kor2idx.get(tok, kor2idx[unk]) for tok in kor_tokens]])\n",
    "    \n",
    "    # starting outputs\n",
    "    decoder_input = np.zeros(shape=(1, 16))\n",
    "    decoder_input[:,0] = eng2idx[sos]\n",
    "    \n",
    "    # greedy decoding\n",
    "    for i in range(maxout-1):\n",
    "        output, heatmap = heatmapmodel.predict([encoder_input, decoder_input])\n",
    "        output = output.argmax(axis=2)\n",
    "        decoder_input[:,i] = output[:,i]\n",
    "\n",
    "    def decode(idxes, idx2tok, pad = '_PAD_', unk = '_UNK_', sos='Ⓑ'):\n",
    "        toks = []\n",
    "        for lst in idxes:\n",
    "            sent = []\n",
    "            for idx in lst:\n",
    "                if idx2tok[idx] not in (pad, unk, sos):\n",
    "                    sent.append(idx2tok[idx])\n",
    "            toks.append(sent)\n",
    "        return toks\n",
    "    \n",
    "    eng_toks = decode(output, idx2eng)[0]\n",
    "    \n",
    "    # trim heatmap\n",
    "    kortrim = maxlen - len(kor_toks)\n",
    "    engtrim = len(eng_toks)\n",
    "    heatmap = heatmap[0]\n",
    "    heatmap = heatmap[:engtrim, kortrim+1:]\n",
    "    print('input :', kor_string)\n",
    "    print('output:', ''.join(eng_toks))\n",
    "    \n",
    "    return eng_toks, kor_toks, heatmap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def heatmap(e, k, a):\n",
    "    krfont = {'family' : 'nanumgothic', 'weight' : 'bold', 'size'   : 16}\n",
    "    matplotlib.rc('font', **krfont)\n",
    "    plt.rcParams[\"figure.figsize\"] = (8,8)\n",
    "\n",
    "    fig, ax = plt.subplots()\n",
    "    im = ax.imshow(a)\n",
    "    ax.set_xticks(np.arange(len(k)))\n",
    "    ax.set_yticks(np.arange(len(e)))\n",
    "    # because single graphemes don't size correctly, add a full character\n",
    "    ax.set_xticklabels([':'+c for c in k[1:]])\n",
    "    ax.set_yticklabels(e)\n",
    "    ax.xaxis.set_ticks_position('top')\n",
    "    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"left\", rotation_mode=\"anchor\")\n",
    "    fig.tight_layout()\n",
    "    plt.show()\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## tests\n",
    "\n",
    "...looks pretty good!\n",
    "\n",
    "notice that in 'annyeong', the initial 'null' consonant 'ㅇ' contributes little to the initial 'a' prediction; instead the network is focusing on the vowel 'ㅏ'. conversely, the single final consonant 'ㅇ', representing a nasal '-ng' sound, is focused on when generating both the 'n' and 'g' of the final *-ng*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input : 안녕\n",
      "output: annyŏng\n",
      "['a', 'n', 'n', 'y', 'ŏ', 'n', 'g']\n"
     ]
    }
   ],
   "source": [
    "e, k, a = translate('안녕')\n",
    "print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf0AAAIrCAYAAADlQ0AnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAFi1JREFUeJzt3X+M5Hd93/HX2z7/BAzYQIhETIUEJi7gNCBCbAgkLaEkIkpQ2oQoNCKYo1UoP6q2FKRGjUVVhISKIqqkhwnhTGJI2oQoJa1jQiENAcXnhKLIIXGtlgrUpAGaHGD7uPN++sfM4fH5bvdu57v7Hfx+PKTVrL87O/P2RzPz3O93v7NXY4wAAA995809AACwP0QfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH2AmVXV+XPPQA+iDzCTqnpqVV0yxrhP+NkPog8wg6p6UpI7knxK+Nkvor+Nqmq5PlX1pKq6ZO454CHuL5N8OMmTk/xuVV0q/Oeu6+v0blX3f1q3qmqMMarqyUmekuSbk3xujPFfVr8+65D7qKquSvInSX47yY+MMf56ub3VOsBeqqrzl4G/PMmNSX4wye1Jnj/GuPvk1+edcnN4nZ5O++gnSVX9cJJ/l+SxK5vfMcb4JzONNJuqemqSDyR5epJfS/LKk+Fffv28McbWXPN9IzjdC5B1u5/1Waiqi8YYx5af35HkqRH+M/I6PY22h0WqqpaXP5HkV5J8Oskbk7wiyZeTvKKq/tZ8E+6/5YvxZ5K8LMnvJ3lpkl+oqguWX39bkoMn144HWw1aVX1XVb0uSboF7Uysz/1Wgv+mLIL/P5I8Mw71f53X6T0wxmj7keQ7k3wtyU1JrlrZ/o4k9yR59twzzrAmJ4/+XJ3k95JsJbk5yX9cfv6GJJfMPeemfSQ5/+TaLf/7+Svr94y555v7w/qccV1uWK7B25J8e5LDy/8+kuTSk2s395wzr5HX6Qk/Wu7pV9V5y73XH19u+sUxxp+uXOXxy8u793ey+Y0xxnJv7I4kB7PY4/+RJD+UxaG1940x7plzxk1RVa+tqvckyRjjvrF8Jaqq5yd5S5Jrk7x0jPHpGcecjfXZXlVdnOSrWTyv3jvG+MMkr8viBL9vT/M9fq/Te2Tunzrm/EjyoSRfPGXbC7P4qfIdc88389qc3ON/WpKPZ7H38Z/SfK9jZX0eluTzy3V598r278n9e7A/eHIts7KX2+HD+pz1Ol2c5FHLzy9ars3RJLfFHv/JNfI6PeFH1z3986vqoiT3JXl0Vf18VT2vql6TRdg+kcWhpBZO/R398qSqk3v8f5zk1Vns8X9fkg9U1aPmmHOTjDG+muS6JJ/N4veK76qqFyT56dy/B/vBk2s7lq9UXVifBzrNc+z8JBlj3DvG+Kvl5v+WxaHs1yV5SRYn1J7c42/3Pn6v03ujxdn7VfXwLE7++LMxxk0r2/9mkt9J8rgsfqo+L8kHk7x9jPHxOWbdb6ecWHXVeODhs9W3ylyd5FAWL9gPOqv/oe7Ux9DKW66uzGLv7AlZnFj0iDQLWmJ9trPTc2y5/dFJrs/id9SHxxhHlz9c35jFCbV3Jvm28RD+1ZrX6X0y96GG/fjI4uSzrSS/meThy23nLS+fkuTfJnlnkh9O8piV72tzyDHJv0nyv5K86DRfO93Jff8hySPnnnvmx9D5y8snZrFHu5XkJSfXrNnjx/rsvEZnfI4tv35xkgPLzy9cXl6W5Jbl2r1g7v+HGR5DXqcn/jhw9j8efEP79SweNO8ZY3wlWbxFaHkY+8+yeLA9QKc/9lBVj0jyvCRXJrn31K+Pcf/JfVV1MIs9/pcmubCqfnyMcXR/J57F6R5D9y33aD9bVc9O8vgxxn/vsgd7CuuzjZ2eY8niUP/K519brt3R5fvTnzrGuG1/pp2N1+l90OLwfpJU1YExxomTfwTklENu7f4wyKmq6vFJnjDGOLLNdVYP9R/O4veNB8cYN+7XnHM602OoTvkjKl1fiKzP9s7mOXaa72n1B3q8Tu+9NtHfSdcXotOpxd+yHqvrcZoX8m9N8neT3DzG+PPZht0gHkPbsz732+45NuNYG89jaH0tz95Pkqp6XFX9bFW9POl1qPEsjJWfrp+x3LB18gvLJ96fJPnZzsH3GNqe9dnWGZ9j3M9jaHpto5/kHyV5TZLvq8UfyWBp5cXo3yf5rar626f7eqfDjmfgMbQ963MGOz3H+DqPoYl1OZHvdN6bxZmxP796Ag0Ly7cLPSPJY5J8aeZxNpXH0PaszzY8x86Kx9DEWv9Ov9tJMueqqh6X5PKx+Ed4OA2Poe1Zn+15ju3MY2haraPP2XMCDewtzzH2g+gDQBOdT+QDgFZEHwCaEP0kyz8tyzas0fasz/asz/asz86s0fbOdn1Ef8GDaWfWaHvWZ3vWZ3vWZ2fWaHuiDwDcb5az9y+si8fF9bB9v98zOT7uzQUb9MeenvC0L889woP81Ze28qjLN+NnxM/f8ci5R3iQr417cmFdMvcYXzfu26y3NR/PsVyQi+YeY2NZn51t2ho95ZlPmnuEB7j99tu/MsZ4xE7XmyX6l513xXjORS/e9/v9RvG2P/3Y3CNstH/xbS+ae4SNd99fd/jXjtfgrcqs6datX517hAeoqtvHGM/a6XqbsesGAOw50QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgibWiX1VXVdX7q+oLVXVvVX2mqv7hVMMBANNZd0//u5O8KMlvJ/m5JBcn+bmq+u5Tr1hVB6vqSFUdOT7uXfNuAYBzdWDN7//NJDeNMb6aJFX1W1n8APDCJP919YpjjENJDiXJZeddMda8XwDgHK0b/S8meU1VvTTJVUkeudz+zWveLgAwsXWj/xtJvjfJ/0zy60lOJHl1klrzdgGAie06+lV1VRbBP5LkuWOMY1X1HVlEHwDYMOucyPew5eWRMcax5ed/f815AIA9ss7h/TuS/EWSf1BVW1n8Hv+HJpkKAJjcrvf0xxj3JnlJkj9K8hNJnp7kDeveLgCwN9Y6kW+McVuS556y+R3r3CYAsDfskQNAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0cWCWex0j49ixWe76G8E/+xvPmXuEjfalV1w99wgb74qX/++5R9ho43s+P/cIMAt7+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATew6+lV1uKq2quqGqrqrqo5W1Uer6popBwQAprHOnv5Wkkry5iSfTHJzkmuTfKiqLj31ylV1sKqOVNWR4zm2xt0CALtxYILbeOMY4+1JUlV3J3l9kuuS3Lp6pTHGoSSHkuSyunxMcL8AwDmY4nf6H1n5/I7l5ZUT3C4AMKEpon/PyufHl5cXTHC7AMCEnL0PAE2IPgA0IfoA0MQ60T+xvDy+wzYAYAPsOvpjjOvHGDXGuGtl2/uW2949zXgAwFQc3geAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJo4MPcAcK6uuOm2uUfYeD/5pjvnHmGjvedpL557hI229cefmXsE9og9fQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJnYd/ao6XFVbVXVDVd1VVUer6qNVdc2UAwIA01hnT38rSSV5c5JPJrk5ybVJPlRVl5565ao6WFVHqurI8Rxb424BgN04MMFtvHGM8fYkqaq7k7w+yXVJbl290hjjUJJDSXJZXT4muF8A4BxM8Tv9j6x8fsfy8soJbhcAmNAU0b9n5fPjy8sLJrhdAGBCzt4HgCZEHwCaEH0AaGKd6J9YXh7fYRsAsAF2Hf0xxvVjjBpj3LWy7X3Lbe+eZjwAYCoO7wNAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0cWDuAeBcjRMn5h5h4/3C1U+ee4SN9q/vPDz3CBvtp6/9gblHYI/Y0weAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaWCv6VfX+qtqqqqetbHtrVY2q+v71xwMAprLunv4vJ6kkL0uSqqokP5bk/ya5ZfWKVXWwqo5U1ZHjObbm3QIA52rd6P/nJF9I8qPL/35ukm9J8stjjBOrVxxjHBpjPGuM8awLctGadwsAnKu1oj/GOJ7k/UmeVFXfkeUef5L3rjsYADCtKU7kO7y8fHmSv5fk02OMT01wuwDAhNaO/hjjtiSfSfLqJI+JvXwA2EhTvWXvpiQHkpxI8ksT3SYAMKGpov8ry8tbxhh/MdFtAgATmir61ywvHdoHgA11YJ1vrqpnJ3lBklcl+VySD04wEwCwB9bd078myVuSjCQvW76FDwDYQGvt6Y8x3pXkXRPNAgDsIf/gDgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0cWDuAYDpjRMn5h5hoz3zogvnHmGj3feFL809AnvEnj4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADRxYKcrVNVbk3wiycPHGL+09yMBAHvhrPb0xxi/keTRVXXt6vaqevqeTAUATO6sD++PMd6Z5MVV9cQkqapvSnL9Xg0GAEzrXH+n/zNJXltVj03yr5K8+Wy/saoOVtWRqjpyPMfO8W4BgHXt+Dv9VWOME1X1liR/mOSFY4yvnsP3HkpyKEkuq8vHOU0JAKxtN2fv/+Mkr0pycOJZAIA9dE7Rr6pXJvnEGOPDST5QVf90b8YCAKZ21of3q+p7k9QY49YkGWPcXlXfsmeTAQCTOqs9/aq6Osl1Y4wbV7ePMT64J1MBAJM7mz39K5L8VJLX7vEsAMAeOpvoP315vd+pqgd9cYzxgolnAgD2wI7RH2M8Zz8GAQD2ln9wBwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaOLA3AMAe6Bq7gk22oue8My5R9hot3zuD+YegT1iTx8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoIldR7+qDlfVVlXdUFV3VdXRqvpoVV0z5YAAwDTW2dPfSlJJ3pzkk0luTnJtkg9V1aWnXrmqDlbVkao6cjzH1rhbAGA3DkxwG28cY7w9Sarq7iSvT3JdkltXrzTGOJTkUJJcVpePCe4XADgHU/xO/yMrn9+xvLxygtsFACY0RfTvWfn8+PLyggluFwCYkLP3AaAJ0QeAJkQfAJpYJ/onlpfHd9gGAGyAXUd/jHH9GKPGGHetbHvfctu7pxkPAJiKw/sA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQhOgDQBOiDwBNiD4ANCH6ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATRyYewBgD4wx9wQbrc6vuUfYaF/ZunfuETbeZXMPsEv29AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4Am1op+Vf1UVd1ZVceq6lNVdUNVjap601QDAgDT2HX0q+r6JO9M8vAkNyb5cpJ/ufzyRae5/sGqOlJVR47n2G7vFgDYpQNrfO8bkmwl+a4xxp1VdV6SjyZ53umuPMY4lORQklxWl4817hcA2IVd7elX1cOSfGuS28YYdybJGGMryeEJZwMAJrTbw/uPTlJJ/s8p2z+33jgAwF7ZbfSPLi+vOGX749aYBQDYQ7uK/hjjaJI/SPKdVfWUJKmq85P85PIqW9OMBwBMZZ237P3zLA7x/25VvTPJ7yd5/vJr/2/dwQCAae06+mOMjyX5O0n+PMkrs3gnwMeWX/699UcDAKa067fsVdUPJHlVkg8nuSXJ1Um+P8mvjTH+aJrxAICprPM+/S8m+aYsDulfksWZ+29N8jMTzAUATGzX0R9jfDzJsyecBQDYQ/7BHQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoQvQBoAnRB4AmRB8AmhB9AGhC9AGgCdEHgCZEHwCaEH0AaEL0AaAJ0QeAJkQfAJoQfQBoosYY+3+nVX+Z5LP7fscA8ND0xDHGY3e60izRBwD2n8P7ANCE6ANAE6IPAE2IPgA0IfoA0IToA0ATog8ATYg+ADQh+gDQxP8HUoN4rTjm1QAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd75c6d95c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "heatmap(e, k, a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input : 김치\n",
      "output: kimch'i\n",
      "['k', 'i', 'm', 'c', 'h', \"'\", 'i']\n"
     ]
    }
   ],
   "source": [
    "e, k, a = translate('김치')\n",
    "print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbQAAAIrCAYAAACH7GxNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAE6NJREFUeJzt3XuMpXddx/HPtzsLpRSUlUsphSoYioCIYUFqQTASQCiQYFJDECVRNlQIEQSCEDGoUSMqV2NZJCKVi2i4SCEChaCiiaSACEZuKlCkVRBCLELZy88/5mw53W5np3POzDP7Pa9XMjkz55x5nl+fPr9573PZ2RpjBABOdadNPQAAWAZBA6AFQQOgBUEDoAVBA6AFQQOgBUEDoAVBA6AFQQOgBUEDoAVBA6AFQQOgBUGbqao67uuV3jZVdVZV3X7qcZzKjt+nYJ45tnwr/UP7mKqqMcaoqvOr6ulJMsY4uqpRq6p7JPnXJL9cVWdPPZ5TwbF4VdV9q+rCJJntU6I2c2w+zc8rc8wcW6aV3JnmzcXs0Un+PsnLq+qlyUpH7U5Jrkryi0measJtbG4fekySjyR5TVU9JxG1Y6rqwUmeWFU3S3K72dcZYxyddmSTMce2Qa3yv4c294PowiR/leQzSe6Q5NZJ/mCM8ezZ+05btYlXVY9K8mtJfijJ7ya5ZIzxpWlHtfscF7O3JflUknOS7EnygjHGS+ffN+FQJ1NVD0vynCRnJXl5kscmuWuSHxtjfK2qbjHG+OaqbSNzbPlW8egjyQljdkWSJyQ5P8k1SZ5VVS9PVutPkceOJsYY70ry60k+luS58afIGzguZm9PcnmSn01y0ewtL6qqZyTrR2oTDXNSVfWIJM9I8oNJ/jLJjyZ5TNaD9q2qelCSt1XV96/KNjLHttEYY+U+8p0j0wuTHE3yoSQPm3v9h5Mcmr32yKnHO9X2mX3+6CT/mORbWZ98Z089vt3wMbcPPWa2n7wnyQPnXn/O7PnLk+yberwTbaM7JrksydeTPC/rQTuS5H+SfDbJryb5ZpKPrto8M8e252NtqXU8RYwxRlU9PsmfZ/3I7PljjMuTpKr2jjE+WlX3z/oPq3+ecKiTmG2fGuveOfsD5Quz/qfIVNXKnxqZbaMH5ztHZs8aY3wiSarqR5I8ZPbWl4wxvjrRMCcz23+uqqrXJ/lgkguS3D/JzyT5lST3TvKiJL+f5A1jjI9ONtgJmGPbY+WCVlV7kuxN8rIk/5HkeWOM989eqzHGoaraM8b4p6r6xBjj8JTjnYoJd+NmNwpVksfPnrrlXMwuSPKCJI9M8vjZtlu1a0Pz/71XJzmQ5F5Jzsv6TRD3SvLmJK8cY3zwRr6vPXNs+Vb2ppCqOivJXccY/zD7eqUm07zj/9vnb4KZf212J+gL4yJ2kuv2obckeWCSd2f9iON5Wb/p4afGGG+du16yUvvWLPq3S/LGrF+XPi/r1+w/luS9SZ4xxvjSqsw7c2xnrMRNIVV1ZlX9RlU9afZ1jTGuFrMbTKbzkuvfBHPsT5Gzz9+ZFb2IfYJ96LQxxtVZP0r7uySPSPL+rGjMjt8+s33o/5K8Icm9xhhfyPrNIbdK8qpjP6RXZNuYYztlpy7WTfmR5JlZv0D/jiRnTj2e3fiR5LeTfC7JI27k9RNdxP7G7PvuOPX4d2D73GAfSrJn9njHJB/I+s0Ojzm2vea3WfePG5tjc9vooUmuzfppxsnHO9E2Mse2+WNVrqG9Ncndk/zJGOOaqQez21TVrZI8OMldsn6n1Q2McYPz/SPJbyV5WpK/qaqrx2wmNnWDfWiMcWR2vfWq2U1Gtx1jfHqVjszmnHCOzbbRnZK8Ismnk7x2muFNyxzbGStzDa2q1sYYh4+duz7uNMDKnnI8ZnY96JwxxhUned/8dvvJrF/gf90Y4793YJiTurF9qFbwL96fyAbb55ys3xjytTHGS6Ye51TMse23MkE7GVH7jtkF/TG/PTa4iL13jHFooqHuKvahzbGdzLHtshI3hZxIVd2+ql4+dxF7pSfYccbcZLrP7InrXcSe+3xlJ5p9aGOz7fOKY9vnGNspiTm2LVY2aEkuTvL0JI+qqtOnHsxuMjfRXpXkXVX1ExMPabeyD23s4qxf/7F9jmOObY9VuSnkRP4067+E+JIxxgkv0q6yqvruJPdJctskK/ebLjbJPrQx22cD5tjyrfQ1tNkdakemHsduVev/+OC+McYnpx7LbmUf2pjtszFzbLlWOmhsngv5sL3MscUJGgAtrPJNIQA0ImgAtCBoSarqwNRj2M1sn43ZPidnG23M9tnYZrePoK2zM23M9tmY7XNyttHGbJ+NCRoAq2OSuxxvs++0cfY5u+fvdH/tq0dzm327p+1XfuJWUw/heg6Nb2XvbvtFD7vo7txDuTZ7c/Oph7Gr2UYb243b5+73u+vUQ7jOhz/84WvGGCf9wThJVc4+Zy1vvux2U6z6lPDM83586iHseuPaa6ceArT23iv+YuohXKeqPrWZ9+2ewxIAWICgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdDCloNWVZ+tqs8uczAAsFVrE30vACyVU44AtCBoALSwtKBV1VOr6khV/dGylgkAm7WUoFXVRUn+MMlfJ3n6jbznQFVdUVVXfO2rR5exWgC4zsJBq6qHJ7k0yUeSXDTGOHKi940xDo4x9o8x9t9mnzOdACzXomX5niRvSfL1JI8eY3xj8SEBwE23aNDOTHLLJLdNcs/FhwMAW7No0I4kOZDkmiSvrapbLz4kALjpFg3aF8cYr07yrCTnJnnZ4kMCgJtuKXdnjDH+OMllSZ5cVY9dxjIB4KZYJGhHknx77uunJPlykldX1b6FRgUAN9GWfx/jGONux319dZLbLzwiANgCfyEMgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFtamWOmVHz8zv3TXB0+x6lPCm7/wgamHsOtddM75Uw8B2GUcoQHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQgqAB0IKgAdCCoAHQwlKDVlUfrKrPV9W5y1wuAJzM2pKX9+IkZyW5asnLBYANLTVoY4y3L3N5ALBZrqEB0MJ2XEM7vMxlAsBmLPsa2lqSPSd6oaoOJDmQJKfnjCWvFoBVt2OnHMcYB8cY+8cY+/fm5ju1WgBWhGtoALQgaAC0IGgAtCBoALQgaAC0sOygHUlydMnLBICTWvavvrpgmcsDgM1yyhGAFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBbWJlvz0SOTrXq3u+ic86cewq532X9+eOoh7GoX3vkBUw9hd/PzpyVHaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtLCpoFXV66pqVNXLquoLVXVNVb2pqvZV1fOr6sqq+t+qem9V/cB2DxoAjre2yfcdnT1enOTPkpye5AlJ9ie5W5LLknw5yc8leUdV3XOM8e35BVTVgSQHkuT0nLH4yAFgzmaDdsyzxhivTJKqOjvJQ5JcMsa4ePbczZI8Mcm9knx0/hvHGAeTHEySW9e+seC4AeB6buo1tL+d+/xzs8fXzz332dnj7bc6IADYipsatG+f4LmvnOC52sJYAGDL3OUIQAuCBkALggZAC5sN2uHZ46FNPnc4ALCDNhW0McYvjDFqjPFvJ3nuN2fPXb4dgwWAG+OUIwAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC2sTT0A2IoL7/yAqYewq1125YemHsKuduGd7jf1ENgGjtAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaGEpQauq+1bV26rqq1V1qKq+UFX7l7FsANiMtUUXUFUXJLk8yZEkb0lyVZK7JPnKce87kORAkpyeMxZdLQBcz8JBS3JJ1o/0HjDG+PiNvWmMcTDJwSS5de0bS1gvAFxnoVOOVXW3JPdOctlGMQOA7bboNbSzZo9XLjoQAFjEokH78uzx+xYdCAAsYtGgfSbJ55I8qqrOX3w4ALA1CwVtjDGSPHO2nA9U1Vuq6neq6mBV3WMpIwSATVj476GNMd6W5OFJPpTkkUmem+RxSb5r0WUDwGYt47b9jDHel+R9y1gWAGyFX30FQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC2tTrLT2rmXttneYYtWnhrVJ/recWqqmHsGuds9Lnz71EHa1cx967dRDYBs4QgOgBUEDoAVBA6AFQQOgBUEDoAVBA6AFQQOgBUEDoAVBA6AFQQOgBUEDoAVBA6AFQQOgBUEDoAVBA6AFQQOgBUEDoAVBA6AFQQOgBUEDoAVBA6AFQQOgBUEDoAVBA6AFQQOgBUEDoAVBA6AFQQOgBUEDoAVBA6CFLQWtql5TVaOq7rzsAQHAVmz1CG3PcY8AMCmnHAFoQdAAaGHRoD27qv69qr5eVZdX1XlLGRUA3ESLBu3JST6Q5I1JHpLk7VXlqA+AHbe24Pf/9BjjnUlSVUeTXJxkf5IPHf/GqjqQ5ECSnL7nzAVXCwDXt+jR1L/Mff7J2eP3nuiNY4yDY4z9Y4z9NzvtFguuFgCub5mnB789ezxjicsEgE1xvQuAFgQNgBYEDYAWthq0I7PHw3PPHTruNQDYMVsK2hjj58cYNcb44txzr5k9d+nyhgcAm+OUIwAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC2sTbHScehwDl/9X1OsGlbCe57wpqmHsKs97fceN/UQ2AaO0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaEHQAGhB0ABoQdAAaGGpQauqJ1XVqKqnLHO5AHAyyz5C23PcIwDsCKccAWhh2UE7Mns8vOTlAsCG1pa5sDHGpUkuPdFrVXUgyYEkOT1nLHO1ALBzpxzHGAfHGPvHGPv35uY7tVoAVoRraAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0IGgAtCBoALQgaAC0sDb1AIDle+q5D5p6CLvau7/0vqmHwDZwhAZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALggZAC4IGQAuCBkALSw1aVX2wqj5fVecuc7kAcDJrS17ei5OcleSqJS8XADa01KCNMd6+zOUBwGa5hgZAC9txDe3wMpcJAJux7Gtoa0n2nOiFqjqQ5ECSnJ4zlrxaAFbdjp1yHGMcHGPsH2Ps35ub79RqAVgRrqEB0IKgAdCCoAHQgqAB0IKgAdDCsoN2JMnRJS8TAE5q2b/66oJlLg8ANsspRwBaEDQAWhA0AFoQNABaEDQAWhA0AFoQNABaEDQAWhA0AFoQNABaEDQAWhA0AFoQNABaEDQAWhA0AFoQNABaEDQAWhA0AFoQNABaEDQAWhA0AFoQNABaEDQAWhA0AFoQNABaEDQAWhA0AFqoMcbOr7Tqy0k+v+MrBuBUdO4Y43Yne9MkQQOAZXPKEYAWBA2AFgQNgBYEDYAWBA2AFgQNgBYEDYAWBA2AFgQNgBb+H1GC9ljwiteAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd7547e1390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "heatmap(e, k, a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "atlas",
   "language": "python",
   "name": "atlas"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}