Seq2Seq神经网络学习.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Seq2Seq 神经网络学习教程\n",
    "\n",
    "本文是对[教程](https://github.com/bentrevett/pytorch-seq2seq)的部分翻译，并加入了一些自己的理解和归纳（Edit by Kevin）\n",
    "<hr>\n",
    "\n",
    "课程中一些注意点：\n",
    "\n",
    "- ①Sequence to Sequence Learning with Neural Networks\n",
    ">使用LSTM；上下文向量$z^n = (h_T^n, c_T^n)$\n",
    "- ②Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation\n",
    ">使用GRU；解码时加入不变的上下文向量(context vector z)\n",
    "- ③Neural Machine Translation by Jointly Learning to Align and Translate\n",
    ">使用双向RNN(GRU);解码时加入Attention\n",
    "- ④Packed Padded Sequences, Masking and Inference\n",
    "- ⑤Convolutional Sequence to Sequence Learning\n",
    "- ⑥Attention is All You Need"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 课程一： Basic学习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 概况\n",
    "\n",
    "在本系列文章中，我们将使用PyTorch和TorchText构建一个机器学习模型，从一个德语序列转换到另一个英语序列。模型可以应用于涉及从一个序列到另一个序列的任何问题，例如翻译、摘要总结。\n",
    "\n",
    "在第一个笔记本中，我们将通过实现[ Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215)来简单地理解一般概念。\n",
    "\n",
    "最常见的seq2seq模型是 *encoder-decoder* 模型, 通常使用一个 *recurrent neural network* (RNN) 去 *编码(encode)* 输入的源文本 (input) 语句成一个向量。在课程一中, 我们将把这个向量称为*上下文向量*。我们可以将上下文向量看作是整个输入语句的抽象表示。然后，第二个RNN对这个向量进行 *解码*。第二个RNN通过每次生成一个单词，来学习去输出目标句子。\n",
    "\n",
    "![](assets/seq2seq1.png)\n",
    "\n",
    "上图显示了一个翻译示例。输入的源语句“guten morgen”，一次输入一个单词到绿色的编码器。 我们还分别在句首和句尾追加了一个*start of sequence* ('<sos\\> ')和*end of sequence* (' <eos\\> ')令牌(Token)。在每一个时间步, 编码器RNN的输入都是当前单词, $x_t$, 以及前一个时间步的隐藏状态, $h_{t-1}$。然后编码器RNN输出一个新的隐藏状态 $h_t$.。你可以把隐藏状态看作是至此句子的所有句的向量表示。这RNN可以表示为 $x_t$ 和 $h_{t-1}$的一个函数:\n",
    "\n",
    "$$h_t = \\text{EncoderRNN}(x_t, h_{t-1})$$\n",
    "\n",
    "我们通常在这里使用术语RNN，它可以是任何递归架构，比如*LSTM* (长短时记忆)或 *GRU* (门控递归单元)。 \n",
    "\n",
    "这里，我们有 $X = \\{x_1, x_2, ..., x_T\\}$, 并且 $x_1 = \\text{<sos>}, x_2 = \\text{guten}$, etc. 初始化的隐藏层状态, $h_0$, 通常要么初始化为零，要么初始化为习得参数。\n",
    "\n",
    "最后一个单词, $x_T$, 被传递到RNN后, 我们使用最后的隐藏层状态, $h_T$, 作为上下文文向量(context vector), i.e. $h_T = z$。 这是整个源语句的向量表示。\n",
    "\n",
    "现在我们有上下文向量, $z$, 我们可以开始解码得到目标句子, \"good morning\"。 同样，我们将开始和结束序列标记附加到目标语句中。在每一个时间步, 解码器RNN(蓝色)的输入是当前单词, $y_t$, 以及前一个时间步长的隐藏状态, $s_{t-1}$, 其中初始解码器隐藏状态, $s_0$, 是那个上下文向量, $s_0 = z = h_T$, i.e. 初始解码器(decoder)隐藏状态为最终编码器(encoder)隐藏状态.。因此，与编码器类似，我们可以将解码器表示为:\n",
    "\n",
    "$$s_t = \\text{DecoderRNN}(y_t, s_{t-1})$$\n",
    "\n",
    "在解码器中, 我们需要将隐藏状态变成一个实际的单词,。因此，在每一个时间步我们使用 $s_t$ 去预测 (通过通过一个线性层`Linear` layer，用紫色表示) 序列中的下一个单词, $\\hat{y}_t$. \n",
    "\n",
    "$$\\hat{y}_t = f(s_t)$$\n",
    "\n",
    "我们总是使用 `<sos>`作为解码器的第一个输入， $y_1$, 但是对于后续的输入, $y_{t>1}$, 我们有时候会用真实的输出单词作为下一个输入单词, $y_t$ 有时候我们使用解码器的输出作为下一个输入单词, $\\hat{y}_{t-1}$。这就是所谓的 *teacher forcing*。\n",
    "\n",
    "我这里简要概括一下*teacher forcing* ：对于模型的预测输出结果，我们计算错误带来的损失后丢弃此输出，而使用真实的输出单词作为下一个输入单词。你可以在[这里](https://machinelearningmastery.com/teacher-forcing-for-recurrent-neural-networks/)了解更多。\n",
    "\n",
    "当训练或测试模型的时候, 我们总是知道目标句中有多少个单词, 所以一旦我们达到这个数量，我们就停止创造单词。 在inference过程中(i.e. real world usage)，通常会一直生成单词，直到模型输出一个“<eos>”Token，或者生成一定数量的单词之后。\n",
    "\n",
    "一旦我们有了预期的目标句, $\\hat{Y} = \\{ \\hat{y}_1, \\hat{y}_2, ..., \\hat{y}_T \\}$, 我们将它与我们的实际目标句进行比较, $Y = \\{ y_1, y_2, ..., y_T \\}$, 计算我们的损失(Loss)。然后，我们使用这个损失来更新模型中的所有参数。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 数据处理\n",
    "\n",
    "我们将在PyTorch中编写模型代码，并使用TorchText帮助我们完成所需的所有预处理。我们还将使用spaCy来辅助数据的标记化。\n",
    "\n",
    "步骤：\n",
    "\n",
    "- 为确定性结果设置随机种子\n",
    "- 使用spaCy创建对应语种的分词器(Tokenizer)，用于将句子转为Token序列，e.g. \"good morning!\" becomes [\"good\", \"morning\", \"!\"]。\n",
    ">spacy_en = spacy.load('en_core_web_sm')\n",
    "spacy_de = spacy.load('de_core_news_sm')\n",
    "- 创建可以传递给TorchText的tokenizer functions来实现分词器功能；这里将输入顺序颠倒，因为实现的论文认为颠倒次序“在数据中引入了许多短期依赖关系，使优化问题变得容易得多”。\n",
    ">def tokenize_de(text):\n",
    "    \"\"\"\n",
    "    Tokenizes German text from a string into a list of strings (tokens) and reverses it\n",
    "    \"\"\"\n",
    "    return [tok.text for tok in spacy_de.tokenizer(text)][::-1]\n",
    "- TorchText的`Field`类处理数据。你可以在[这里](https://github.com/pytorch/text/blob/master/torchtext/data/field.py#L61)阅读所有可能的函数参数。\n",
    ">SRC = Field(tokenize=tokenize_de, init_token='<sos>', eos_token='<eos>', lower=True)\n",
    "TRG = Field(tokenize=tokenize_en, init_token='<sos>', eos_token='<eos>', lower=True)\n",
    "\n",
    "- 我们下载并加载训练、验证和测试数据。我们将使用的数据集是Multi30k数据集。这是一个包含约30000个平行的英语、德语和法语句子的数据集，每个句子包含约12个单词。`exts`指定使用哪种语言作为source和target(source goes first)，`fields` 指定使用哪个fields 为 source和target所使用。\n",
    ">train_data, valid_data, test_data = Multi30k.splits(exts=('.de', '.en'), fields=(SRC, TRG))\n",
    "\n",
    "- 为源语言和目标语言分别构建词汇表，使每个Token对应唯一的一个索引，用于构建最终输出的One-hot向量。注：词汇表只从训练集构建，而不是验证集或测试集\n",
    ">SRC.build_vocab(train_data, min_freq=2)<br>\n",
    "TRG.build_vocab(train_data, min_freq=2)<br>\n",
    "//min_freq指定最小频数，低于此数，Token使用 `<unk>` 代替\n",
    "\n",
    "- 创建迭代器，对训练验证测试数据进行迭代，以返回一批具有`src`属性和`trg`的数据（数据已使用索引代替Token）padding\n",
    ">BATCH_SIZE = 128<br>\n",
    ">device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')<br>\n",
    ">train_iterator, valid_iterator, test_iterator = BucketIterator.splits(\n",
    "    (train_data, valid_data, test_data), batch_size=BATCH_SIZE, device=device)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 构建Seq2Seq模型\n",
    "我们将分三部分构建模型。编码器、解码器和封装编码器和解码器的seq2seq模型，并将提供一种与它们进行交互的方法。\n",
    "\n",
    "## Encoder\n",
    "\n",
    "编码器，使用一个2层LSTM构建。\n",
    "\n",
    "对于多层RNN，输入的句子 $X$ 和隐藏状态 $H=\\{h_1, h_2, ..., h_T\\}$ ，进入多层RNN的底部第一层。该层的输出用作上面层RNN的输入。因此，用上标表示每一层，第一层中的隐藏状态可表示为：\n",
    "\n",
    "$$h_t^1 = \\text{EncoderRNN}^1(x_t, h_{t-1}^1)$$\n",
    "\n",
    "第二层的隐藏状态可表示为：\n",
    "\n",
    "$$h_t^2 = \\text{EncoderRNN}^2(h_t^1, h_{t-1}^2)$$\n",
    "\n",
    "使用多层RNN也意味着我们还需要给每层输入一个初始隐藏状态,$h_0^l$,每层也会输出一个上下文向量, $z^l$。\n",
    "\n",
    "这里不详细介绍LSTMs（如果您想了解关于它们的更多信息，请参阅[本文](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)）。但我们需要知道的是，LSTM不仅在每个时间步中接收并返回隐藏状态，还接收并返回单元格状态 *cell state*, $c_t$：\n",
    "\n",
    "$$\\begin{align*}\n",
    "h_t &= \\text{RNN}(x_t, h_{t-1})\\\\\n",
    "(h_t, c_t) &= \\text{LSTM}(x_t, (h_{t-1}, c_{t-1}))\n",
    "\\end{align*}$$\n",
    "\n",
    "你可以将$c_t$看做另一种隐藏层状态。类似于$h_0^l$， $c_0^l$将初始化为一个所有0的张量。同样，我们的上下文向量现在将是最终的隐藏状态和最终的单元格状态，即$z^l = (h_T^l, c_T^l)$\n",
    "\n",
    "将我们的multi-layer方程推广到LSTMs，得到：\n",
    "\n",
    "$$\\begin{align*}\n",
    "(h_t^1, c_t^1) &= \\text{EncoderLSTM}^1(x_t, (h_{t-1}^1, c_{t-1}^1))\\\\\n",
    "(h_t^2, c_t^2) &= \\text{EncoderLSTM}^2(h_t^1, (h_{t-1}^2, c_{t-1}^2))\n",
    "\\end{align*}$$\n",
    "\n",
    "注意！只有来自第一层的隐藏状态作为输入传递到第二层，单元格状态并没有传递到第二层。\n",
    "\n",
    "我们的编码器是这样的:\n",
    "\n",
    "![](assets/seq2seq2.png)\n",
    "\n",
    "这里不对Embedding和Pytorch的LSTM构建展开过多叙述，详细可以查看Pytorch官方文档和[原教程](https://github.com/bentrevett/pytorch-seq2seq)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.input_dim = input_dim\n",
    "        self.emb_dim = emb_dim\n",
    "        self.hid_dim = hid_dim\n",
    "        self.n_layers = n_layers\n",
    "        self.dropout = dropout\n",
    "        \n",
    "        self.embedding = nn.Embedding(input_dim, emb_dim)\n",
    "        \n",
    "        self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout=dropout)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, src):\n",
    "        \n",
    "        #src = [sent len, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(src))\n",
    "        \n",
    "        #embedded = [sent len, batch size, emb dim]\n",
    "        \n",
    "        outputs, (hidden, cell) = self.rnn(embedded)\n",
    "        \n",
    "        #outputs = [sent len, batch size, hid dim * n directions]\n",
    "        #hidden = [n layers * n directions, batch size, hid dim]\n",
    "        #cell = [n layers * n directions, batch size, hid dim]\n",
    "        \n",
    "        #outputs are always from the top hidden layer\n",
    "        \n",
    "        return hidden, cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoder\n",
    "\n",
    "接下来，我们将构建解码器，解码器也是一个2层LSTM。\n",
    "\n",
    "![](assets/seq2seq3.png)\n",
    "\n",
    "解码器只执行一个解码步骤。第一层将接收前一个时间步的隐藏状态和单元格状态$(s_{t-1}^1, c_{t-1}^1)$，并使用当前Token  $y_t$  一起传递给LSTM，生成一个新的隐藏状态和单元格状态$(s_t^1, c_t^1)$。后面的层将使用他们下面一层的隐藏状态$s_t^{l-1}$，还有前面一层的隐藏状态和单元格状态$(s_{t-1}^l, c_{t-1}^l)$。这提供了与编码器中非常相似的方程。\n",
    "\n",
    "$$\\begin{align*}\n",
    "(s_t^1, c_t^1) = \\text{DecoderLSTM}^1(y_t, (s_{t-1}^1, c_{t-1}^1))\\\\\n",
    "(s_t^2, c_t^2) = \\text{DecoderLSTM}^2(s_t^1, (s_{t-1}^2, c_{t-1}^2))\n",
    "\\end{align*}$$\n",
    "\n",
    "记住，解码器的初始隐藏状态和单元状态是来自同一层的编码器的最终隐藏状态和单元状态，也叫做上下文向量, i.e. \n",
    "\n",
    "$$\\begin{align*}\n",
    "$(s_0^l,c_0^l)=z^l=(h_T^l,c_T^l)$.\n",
    "\\end{align*}$$\n",
    "\n",
    "然后我们把顶层RNN的隐藏层状态, $s_t^L$,传递给线性层 ，$f$，去预测输出的目标序列的下一个Token是什么，$\\hat{y}_{t+1}$。\n",
    "\n",
    "$$\\hat{y}_{t+1} = f(s_t^L)$$\n",
    "\n",
    "Docoder模型查看Pytorch官方文档和[原教程](https://github.com/bentrevett/pytorch-seq2seq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self, output_dim, emb_dim, hid_dim, n_layers, dropout):\n",
    "        super().__init__()\n",
    "\n",
    "        self.emb_dim = emb_dim\n",
    "        self.hid_dim = hid_dim\n",
    "        self.output_dim = output_dim#输出的one-hot大小，起决于词汇表大小\n",
    "        self.n_layers = n_layers\n",
    "        self.dropout = dropout\n",
    "        \n",
    "        self.embedding = nn.Embedding(output_dim, emb_dim)\n",
    "        \n",
    "        self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout=dropout)\n",
    "        \n",
    "        self.out = nn.Linear(hid_dim, output_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, input, hidden, cell):\n",
    "        \n",
    "        #input = [batch size]\n",
    "        #hidden = [n layers * n directions, batch size, hid dim]\n",
    "        #cell = [n layers * n directions, batch size, hid dim]\n",
    "        \n",
    "        #n directions in the decoder will both always be 1, therefore:\n",
    "        #hidden = [n layers, batch size, hid dim]\n",
    "        #context = [n layers, batch size, hid dim]\n",
    "        \n",
    "        input = input.unsqueeze(0)\n",
    "        \n",
    "        #input = [1, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(input))\n",
    "        \n",
    "        #embedded = [1, batch size, emb dim]\n",
    "                \n",
    "        output, (hidden, cell) = self.rnn(embedded, (hidden, cell))\n",
    "        \n",
    "        #output = [sent len, batch size, hid dim * n directions]\n",
    "        #hidden = [n layers * n directions, batch size, hid dim]\n",
    "        #cell = [n layers * n directions, batch size, hid dim]\n",
    "        \n",
    "        #sent len and n directions will always be 1 in the decoder, therefore:\n",
    "        #output = [1, batch size, hid dim]\n",
    "        #hidden = [n layers, batch size, hid dim]\n",
    "        #cell = [n layers, batch size, hid dim]\n",
    "        \n",
    "        prediction = self.out(output.squeeze(0))\n",
    "        \n",
    "        #prediction = [batch size, output dim]\n",
    "        \n",
    "        return prediction, hidden, cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seq2Seq\n",
    "\n",
    "实现Seq2seq模型的最后一部分, 我们将要处理以下问题： \n",
    "- 接收原句输入\n",
    "- 使用编码器生成上下文向量 \n",
    "- 使用解码器生成预测的输出/目标句\n",
    "\n",
    "我们的完整模型将是这样的:\n",
    "\n",
    "![](assets/seq2seq4.png)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "注意，我们必须确保在编码器和解码器中层(layer)数和隐藏状态(和单元格)维数相等。不然多层编码器产生的多个上下文向量无法被一一对应接收。\n",
    "\n",
    "我讲在这部分再回顾一遍整体过程：\n",
    "\n",
    "模型的forward方法喂入源句、目标句和teacher-forcing比率。在对模型进行训练时，采用了 teacher forcing比率。解码时,在每个时间步我们将预测目标序列中的下一个Token将从以前的令牌解码,  $\\hat{y}_{t+1}=f(s_t^L)$ 。当概率等于teacher-forcing比率(teacher_forcing_ratio)时，我们将使用序列中的实际ground-truth next Token作为下一个时间步的解码器输入。但是，对于概率为1 - teacher_forced ing_ratio的情况，我们将使用模型预测的Token。\n",
    "\n",
    "我们在模型forward方法中做的第一件事是创建一个`outputs`张量，它将存储我们所有的预测，$\\hat{Y}$。\n",
    "\n",
    "然后给模型Encoder喂入原句, $X$/`src`, 接收Encoder的最终输出隐藏层状态（hidden states）和单元状态（cell states）。\n",
    "\n",
    "解码器的第一个输入是句子的开始—— (`<sos>`) token。 由于我们的`trg `张量已经附加了` <sos> `token(回溯到我们在`TRG `Field中定义`init_token `的时候)，我们通过切片得到了$y_1$。 我们使用 `max_len`规定句子长度, 我们循环了很多次。在循环的每次迭代中，我们：\n",
    "- 喂入输入数据, 输出 ($y_t, s_{t-1}, c_{t-1}$) 进入解码器\n",
    "- 从解码器接收输出预测、下个隐藏层状态和单元状态 ($\\hat{y}_{t+1}, s_{t}, c_{t}$) \n",
    "- 把我们的预测结果, $\\hat{y}_{t+1}$/`output` 放置入存储预测结果集合的张量Tensor中, $\\hat{Y}$/`outputs`\n",
    "- 决定我们是否要\"teacher force\" ：\n",
    "    - 如果要, 下一个时间步的解码器输入 `input` 使用序列中的实际ground-truth next Token, $y_{t+1}$/`trg[t]`\n",
    "    - 如果不要, 下一个时间步的解码器输入 `input` 使用模型预测的Token, $\\hat{y}_{t+1}$/`top1`\n",
    "    \n",
    "一旦我们做出了所有的预测，我们就会返回一个充满预测的张量, $\\hat{Y}$/`outputs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder, device):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        self.device = device\n",
    "        \n",
    "        assert encoder.hid_dim == decoder.hid_dim, \"Hidden dimensions of encoder and decoder must be equal!\"\n",
    "        assert encoder.n_layers == decoder.n_layers, \"Encoder and decoder must have equal number of layers!\"\n",
    "        \n",
    "    def forward(self, src, trg, teacher_forcing_ratio=0.5):\n",
    "        \n",
    "        #src = [sent len, batch size]\n",
    "        #trg = [sent len, batch size]\n",
    "        #teacher_forcing_ratio is probability to use teacher forcing\n",
    "        #e.g. if teacher_forcing_ratio is 0.75 we use ground-truth inputs 75% of the time\n",
    "        \n",
    "        batch_size = trg.shape[1]\n",
    "        max_len = trg.shape[0]\n",
    "        trg_vocab_size = self.decoder.output_dim\n",
    "        \n",
    "        #tensor to store decoder outputs\n",
    "        outputs = torch.zeros(max_len, batch_size, trg_vocab_size).to(self.device)\n",
    "        \n",
    "        #last hidden state of the encoder is used as the initial hidden state of the decoder\n",
    "        hidden, cell = self.encoder(src)\n",
    "        \n",
    "        #first input to the decoder is the <sos> tokens\n",
    "        input = trg[0,:]\n",
    "        \n",
    "        for t in range(1, max_len):\n",
    "            \n",
    "            output, hidden, cell = self.decoder(input, hidden, cell)\n",
    "            outputs[t] = output\n",
    "            teacher_force = random.random() < teacher_forcing_ratio\n",
    "            top1 = output.max(1)[1]\n",
    "            input = (trg[t] if teacher_force else top1)\n",
    "        \n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练/测试  模型\n",
    "\n",
    "此处基本与通常训练步骤相似，不再展开。\n",
    "\n",
    "\n",
    "注意点：\n",
    "- 损失函数只对2d输入、1d输出的起效，所以计算损失前需要使用`.view`降维\n",
    "\n",
    "- 我们也不想计算序列开头`<sos>`token带来的损失。因此，我们切掉输出和目标张量的第一列\n",
    "\n",
    ">loss = criterion(output[1:].view(-1, output.shape[2]), trg[1:].view(-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 课程二： 进阶学习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 进阶概况\n",
    "\n",
    ">上节课的模型缺点：试图将大量信息塞进隐藏状态。在解码时，隐藏状态将需要包含关于整个源序列的信息，以及到目前为止已解码的所有Token\n",
    "\n",
    "进阶学习：\n",
    "- 针对上面提到的缺点，降低信息压缩的影响\n",
    "- 使用GRU代替LSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 构建模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Encoder\n",
    "\n",
    "该编码器与前一个类似，用单层GRU替换了多层LSTM\n",
    "\n",
    "**具体区别**：\n",
    "- 由于我们只有一个单层，所以我们不对GRU模型进行dropout。因为dropout用于多层RNN的每一层之间。\n",
    "- ：关于GRU需要注意的另一件事是，它只需要并返回一个隐藏状态，没有像LSTM中那样的单元格状态。\n",
    "\n",
    "$$\\begin{align*}\n",
    "h_t &= \\text{GRU}(x_t, h_{t-1})\\\\\n",
    "(h_t, c_t) &= \\text{LSTM}(x_t, (h_{t-1}, c_{t-1}))\\\\\n",
    "h_t &= \\text{RNN}(x_t, h_{t-1})\n",
    "\\end{align*}$$\n",
    "\n",
    "从上面的方程可以看出，RNN和GRU是一样的。然而，在GRU内部有许多门控机制（*gating mechanisms*） ，控制进入和退出隐藏状态的信息流(类似于LSTM)。要了解更多信息请见[此](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)。\n",
    "\n",
    "所以整体流程与上一篇十分类似： 输入序列  $X = \\{x_1, x_2, ... , x_T\\}$， 递归地计算隐藏状态  $H = \\{h_1, h_2, ..., h_T\\}$，然后将最终的隐藏状态作为输出返回  $z=h_T$。\n",
    "\n",
    "$$h_t = \\text{EncoderGRU}(x_t, h_{t-1})$$\n",
    "\n",
    "这与一般seq2seq模型的编码器相同，所有的“魔法”都发生在GRU(绿色方块)中。\n",
    "\n",
    "![](assets/seq2seq5.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, input_dim, emb_dim, hid_dim, dropout):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.input_dim = input_dim\n",
    "        self.emb_dim = emb_dim\n",
    "        self.hid_dim = hid_dim\n",
    "        self.dropout = dropout\n",
    "        \n",
    "        self.embedding = nn.Embedding(input_dim, emb_dim) #no dropout as only one layer!\n",
    "        \n",
    "        self.rnn = nn.GRU(emb_dim, hid_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, src):\n",
    "        \n",
    "        #src = [sent len, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(src))\n",
    "        \n",
    "        #embedded = [sent len, batch size, emb dim]\n",
    "        \n",
    "        outputs, hidden = self.rnn(embedded) #no cell state!\n",
    "        \n",
    "        #outputs = [sent len, batch size, hid dim * n directions]\n",
    "        #hidden = [n layers * n directions, batch size, hid dim]\n",
    "        \n",
    "        #outputs are always from the top hidden layer\n",
    "        \n",
    "        return hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoder\n",
    "\n",
    "解码器的实现与以前的模型有很大的不同，我们减轻了一些信息压缩。\n",
    "\n",
    "解码器中的GRU不仅接受目标Token $y_t$ 和前面的隐藏状态 $s_{t-1}$ 作为输入，它还接受上下文向量 $z$ 。\n",
    "\n",
    "$$s_t = \\text{DecoderGRU}(y_t, s_{t-1}, z)$$\n",
    "\n",
    "现在解码器中的上下文向量没有下标 $t$ ，因为我们再解码器的每个时间步中我们一直使用Encoder产生的那个上下文向量，没有改变~\n",
    "\n",
    "在此之前，我们使用线性层$f$预测下一个Token$\\hat{y}_{t+1}$，只使用当时的解码器顶层隐藏状态$s_t$。\n",
    "\n",
    "$\\hat{y}_{t+1}=f(s_t^L)$\n",
    "\n",
    "现在，我们还将当前Token $\\hat{y}_t$ 和上下文向量$z$传递给线性层。\n",
    "\n",
    "$$\\hat{y}_{t+1} = f(y_t, s_t, z)$$\n",
    "\n",
    "\n",
    "因此，我们的解码器现在看起来是这样的:\n",
    "\n",
    "![](assets/seq2seq6.png)\n",
    "\n",
    "\n",
    "\n",
    "假设解码器隐藏状态,$s_t$,不再需要包含源序列信息,因为它总是可以作为输入。因此，它只需要包含到目前为止生成的 Token 的信息。添加 $y_t$ 线性层也意味着这一层可以直接看到Token是什么,无需得到这个信息隐藏状态。\n",
    "\n",
    "然而，这个假设只是一个假设，不可能确定模型实际上是如何使用提供给它的信息的。然而，这是一个坚实的直觉和结果似乎表明，这种修改是一个好主意!\n",
    "\n",
    "在实际实现中，我们将把 隐藏状态$y_t$ 和 上下文向量 $z$ 连接起来传递给GRU，因此GRU的输入维度现在是  `emb_dim + hid_dim` (上下文向量的大小是' hid_dim ')。线性层将 $y_t, s_t$ 和 $z$ 连接在一起，因此输入维度现在是 `emb_dim + hid_dim*2` 。我们也不会将dropout的值传递给GRU，因为它只使用一个单层。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self, output_dim, emb_dim, hid_dim, dropout):\n",
    "        super().__init__()\n",
    "\n",
    "        self.emb_dim = emb_dim\n",
    "        self.hid_dim = hid_dim\n",
    "        self.output_dim = output_dim\n",
    "        self.dropout = dropout\n",
    "        \n",
    "        self.embedding = nn.Embedding(output_dim, emb_dim)\n",
    "        \n",
    "        self.rnn = nn.GRU(emb_dim + hid_dim, hid_dim)\n",
    "        \n",
    "        self.out = nn.Linear(emb_dim + hid_dim*2, output_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, input, hidden, context):\n",
    "        \n",
    "        #input = [batch size]\n",
    "        #hidden = [n layers * n directions, batch size, hid dim]\n",
    "        #context = [n layers * n directions, batch size, hid dim]\n",
    "        \n",
    "        #n layers and n directions in the decoder will both always be 1, therefore:\n",
    "        #hidden = [1, batch size, hid dim]\n",
    "        #context = [1, batch size, hid dim]\n",
    "        \n",
    "        input = input.unsqueeze(0)\n",
    "        \n",
    "        #input = [1, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(input))\n",
    "        \n",
    "        #embedded = [1, batch size, emb dim]\n",
    "                \n",
    "        emb_con = torch.cat((embedded, context), dim=2)\n",
    "            \n",
    "        #emb_con = [1, bsz, emb dim + hid dim]\n",
    "            \n",
    "        output, hidden = self.rnn(emb_con, hidden)\n",
    "        \n",
    "        #output = [sent len, batch size, hid dim * n directions]\n",
    "        #hidden = [n layers * n directions, batch size, hid dim]\n",
    "        \n",
    "        #sent len, n layers and n directions will always be 1 in the decoder, therefore:\n",
    "        #output = [1, batch size, hid dim]\n",
    "        #hidden = [1, batch size, hid dim]\n",
    "        \n",
    "        output = torch.cat((embedded.squeeze(0), hidden.squeeze(0), context.squeeze(0)), dim=1)\n",
    "        \n",
    "        #output = [batch size, emb dim + hid dim * 2]\n",
    "        \n",
    "        prediction = self.out(output)\n",
    "        \n",
    "        #prediction = [batch size, output dim]\n",
    "        \n",
    "        return prediction, hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seq2Seq Model\n",
    "\n",
    "将编码器和解码器放在一起，我们得到\n",
    "\n",
    "![](assets/seq2seq7.png)\n",
    "\n",
    "同样，在这个实现中，我们需要确保编码器和解码器中的隐藏状态维度是相同的。\n",
    "\n",
    "简单回顾一下所有步骤:\n",
    "- 输出张量 `outputs` 用于保存所有预测结果, $\\hat{Y}$\n",
    "- 源序列, $X$, 输入编码器以获得上下文向量\n",
    "- 解码器的初始隐藏状态为上下文状态, $s_0 = z = h_T$\n",
    "- 我们使用一批 `<sos>` tokens 作为首先的输入`input`, $y_1$\n",
    "- 然后我们在一个循环中解码:\n",
    "  - 将输入的Token $y_t$, 之前的隐藏层状态, $s_{t-1}$, 和上下文向量, $z$, 输入解码器\n",
    "  - 接收一个预测, $\\hat{y}_{t+1}$, 和一个新的隐藏层状态, $s_t$\n",
    "  - 然后，我们决定是否要 `teacher force` , 根据需要设置下一个输入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder, device):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        self.device = device\n",
    "        \n",
    "        assert encoder.hid_dim == decoder.hid_dim, \"Hidden dimensions of encoder and decoder must be equal!\"\n",
    "        \n",
    "    def forward(self, src, trg, teacher_forcing_ratio=0.5):\n",
    "        \n",
    "        #src = [sent len, batch size]\n",
    "        #trg = [sent len, batch size]\n",
    "        #teacher_forcing_ratio is probability to use teacher forcing\n",
    "        #e.g. if teacher_forcing_ratio is 0.75 we use ground-truth inputs 75% of the time\n",
    "        \n",
    "        batch_size = trg.shape[1]\n",
    "        max_len = trg.shape[0]\n",
    "        trg_vocab_size = self.decoder.output_dim\n",
    "        \n",
    "        #tensor to store decoder outputs\n",
    "        outputs = torch.zeros(max_len, batch_size, trg_vocab_size).to(self.device)\n",
    "        \n",
    "        #last hidden state of the encoder is the context\n",
    "        context = self.encoder(src)\n",
    "        \n",
    "        #context also used as the initial hidden state of the decoder\n",
    "        hidden = context\n",
    "        \n",
    "        #first input to the decoder is the <sos> tokens\n",
    "        input = trg[0,:]\n",
    "        \n",
    "        for t in range(1, max_len):\n",
    "            \n",
    "            output, hidden = self.decoder(input, hidden, context)\n",
    "            outputs[t] = output\n",
    "            teacher_force = random.random() < teacher_forcing_ratio\n",
    "            top1 = output.max(1)[1]\n",
    "            input = (trg[t] if teacher_force else top1)\n",
    "\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型训练 / 测试\n",
    "\n",
    "<hr>\n",
    "不再展开拉，请参照原教程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 课程三： 初识Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 课程概况\n",
    "\n",
    ">上节课的模型缺点：即使我们减少了一些压缩，上下文向量仍然需要包含关于源语句的所有信息\n",
    "\n",
    "**进阶学习：**\n",
    "\n",
    "在本课程中实现的模型的解码器，可在每个解码步骤中通过其隐藏状态来查看整个源语句，来避免这种压缩!\n",
    "\n",
    "它是怎么做到的?\n",
    "\n",
    "它使用的即是 `Attention` 机制。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Attention简介\n",
    "\n",
    "Attention机制\n",
    "\n",
    "- 首先对于注意力**编码器RNN**的所有隐藏状态，计算一个softmax后的注意力权重向量，$a$，向量维度等于原序列的长度。\n",
    "- 然后我们计算源语句隐藏状态的加权和，得到一个加权的源向量 $w$ 。在解码过程中，每一步都计算一个新的加权源向量，并将其作为解码器RNN和线性层的输入进行预测\n",
    "\n",
    "$$w = \\sum_{i}a_ih_i$$\n",
    "\n",
    "<hr>\n",
    "\n",
    "相信一开始接触的时候对这个概念还是有点不明白的，这里搬运[表哥博客](http://jun369.me)里的[文章](http://jun369.me/2019/01/28/Deep%20Learning/Natural%20Language%20Processing/%E5%8F%AF%E8%A7%86%E5%8C%96seq2seq/)对此可视化的解释,想了解更多请参考原文哈：\n",
    "\n",
    "- 第一，经典的sequence-to-sequence模型的解码器只传递了最后一个隐藏状态而注意力模型的编码器将所有的隐藏状态传到解码器：\n",
    "\n",
    "![](./assets/seq7.gif)\n",
    "\n",
    "- 第二, 一个有注意力的解码器需要在它产生输出前能够关注到跟这个时间步有关的信息，所以它需要多做一步:\n",
    "\n",
    ">观察一下编码器送来的隐藏状态，每一个隐藏状态都和句子中的某个单词相对应。\n",
    "\n",
    ">给每一个隐藏状态一个分数(我们先忽略分数是如何给出的)。\n",
    "\n",
    ">把分数进行softmax之后与对应的隐藏状态相乘，从而放大分数高的隐藏状态，缩小分数低的隐藏状态。\n",
    "\n",
    "![](./assets/attention-process.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 模型构建"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder\n",
    "\n",
    "首先，我们将构建编码器。与之前的模型类似，我们只使用单层GRU，但是现在我们使用双向RNN *bidirectional RNN*。\n",
    "对于双向RNN，我们在每一层都有两个RNN。\n",
    "一个*forward RNN* 从左到右浏览这个句子 (下图为绿色), 和一个 *backward RNN* 一个从右到左的倒向RNN(下图为黄色)。\n",
    "\n",
    "![](assets/seq2seq8.png)\n",
    "\n",
    "即:\n",
    "\n",
    "$$\\begin{align*}\n",
    "h_t^\\rightarrow &= \\text{EncoderGRU}^\\rightarrow(x_t^\\rightarrow,h_t^\\rightarrow)\\\\\n",
    "h_t^\\leftarrow &= \\text{EncoderGRU}^\\leftarrow(x_t^\\leftarrow,h_t^\\leftarrow)\n",
    "\\end{align*}$$\n",
    "\n",
    "其中 $x_0^\\rightarrow = \\text{<sos>}, x_1^\\rightarrow = \\text{guten}$ ， $x_0^\\leftarrow = \\text{<eos>}, x_1^\\leftarrow = \\text{morgen}$.\n",
    "\n",
    "类似于之前， 我们向RNN只传递一个输入(`embedded`) to the RNN, 使PyTorch同时初始化正向和反向两个初始隐藏状态 ($h_0^\\rightarrow$ and $h_0^\\leftarrow$) 为两个全0张量。 我们还会得到两个上下文向量，一个来自于前向RNN（当它看到句子中的最后一个单词后产生）， $z^\\rightarrow=h_T^\\rightarrow$, 一个来自于后向RNN（在它看到句子中的第一个单词后产生）， $z^\\leftarrow=h_T^\\leftarrow$.\n",
    "\n",
    "RNN 返回输出 `outputs` 和隐藏状态 `hidden`. \n",
    "\n",
    "输出`outputs` 的大小为 **[sent len, batch size, hid dim * num directions]** 其中第三维的 `hid_dim` 来自`forward RNN`和`backward RNN`的隐藏状态大小。你可以把第三个轴想象成正向和反向隐藏状态叠加在一起，i.e. $h_1 = [h_1^\\rightarrow; h_{T}^\\leftarrow]$, $h_2 = [h_2^\\rightarrow; h_{T-1}^\\leftarrow]$ ，我们可以把所有堆叠的编码器隐藏状态表示为 $H=\\{ h_1, h_2, ..., h_T\\}$.\n",
    "\n",
    "隐藏状态`hidden` 的大小为 **[n layers * num directions, batch size, hid dim]**, 其中 **[-2, :, :]** 在经过最后的时间步后给出顶层  'forward RNN'  隐藏状态 (i.e. after it has seen the last word in the sentence) 还有 **[-1, :, :]** 在经过最后的时间步后给出顶层  'backward RNN'  隐藏状态  (i.e. after it has seen the first word in the sentence).\n",
    "\n",
    "因为解码器不是双向的, 它只需要一个上下文向量, $z$, 用作它的初始隐藏状态, $s_0$。但我们现在有两个上下文向量($z^\\rightarrow=h_T^\\rightarrow$ and $z^\\leftarrow=h_T^\\leftarrow$, respectively)。为解决这个问题，我们连接两个上下文向量,使它们通过一个线性层,$g$,应用双曲正切激活函数$\\tanh$产生最终结果。 \n",
    "\n",
    "$$z=\\tanh(g(h_T^\\rightarrow, h_T^\\leftarrow)) = \\tanh(g(z^\\rightarrow, z^\\leftarrow)) = s_0$$\n",
    "\n",
    "**Note**:这实际上是偏离了所参考的论文。相反，它们只向线性层提供第一个后向RNN隐藏状态来获得上下文向量/解码器的初始隐藏状态。这对本课程来说似乎没有意义，所以本课程改变了它。\n",
    "\n",
    "当我们希望我们的模型回顾整个源语句时，我们返回`outputs`，即源语句中每个Token的前向和后向隐藏状态。我们还返回`hidden`，它作为我们的初始隐藏状态在解码器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, input_dim, emb_dim, enc_hid_dim, dec_hid_dim, dropout):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.input_dim = input_dim\n",
    "        self.emb_dim = emb_dim\n",
    "        self.enc_hid_dim = enc_hid_dim\n",
    "        self.dec_hid_dim = dec_hid_dim\n",
    "        self.dropout = dropout\n",
    "        \n",
    "        self.embedding = nn.Embedding(input_dim, emb_dim)\n",
    "        \n",
    "        self.rnn = nn.GRU(emb_dim, enc_hid_dim, bidirectional=True)\n",
    "        \n",
    "        self.fc = nn.Linear(enc_hid_dim * 2, dec_hid_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, src):\n",
    "        \n",
    "        #src = [src sent len, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(src))\n",
    "        \n",
    "        #embedded = [src sent len, batch size, emb dim]\n",
    "        \n",
    "        outputs, hidden = self.rnn(embedded)\n",
    "                \n",
    "        #outputs = [sent len, batch size, hid dim * num directions]\n",
    "        #hidden = [n layers * num directions, batch size, hid dim]\n",
    "        \n",
    "        #hidden is stacked [forward_1, backward_1, forward_2, backward_2, ...]\n",
    "        #outputs are always from the last layer\n",
    "        \n",
    "        #hidden [-2, :, : ] is the last of the forwards RNN \n",
    "        #hidden [-1, :, : ] is the last of the backwards RNN\n",
    "        \n",
    "        #initial decoder hidden is final hidden state of the forwards and backwards encoder RNNs fed through a linear layer\n",
    "        hidden = torch.tanh(self.fc(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim=1)))\n",
    "        \n",
    "        #outputs = [sent len, batch size, enc hid dim * 2]\n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        \n",
    "        return outputs, hidden"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attention\n",
    "\n",
    "接下来是注意力层。这将接受解码器的前一个隐藏状态$s_{t-1}$，以及编码器的所有前向隐藏状态和后向隐藏状态的集合(stack), $H$。 这个线性层会输出一个注意向量, $a_t$, 其维度大小即是源语句长度, 每个元素都在0到1之间，整个向量和为1。\n",
    "\n",
    "直观地说，这一层使用我们目前已经解码的 $s_{t-1}$ 和所有已经编码的 $H$ 来生成一个向量 $a_t$ ，它表示源语句中我们应该最注意哪些单词，以便正确地预测下一个要解码的单词$\\hat{y}_{t+1}$。\n",
    "\n",
    "首先，我们计算前一个解码器隐藏状态和编码器隐藏状态之间的`energy`。 由于我们的编码器隐藏状态是一个由 $T$ 张量组成的序列，而我们之前的解码器隐藏状态只是一个张量，所以我们要做的第一件事就是“重复”之前的解码器隐藏状态$T$次。 然后计算 `energy`, $E_t$, 通过把他们连接起来，并通过一个线性层(`attn`)和一个$\\tanh$ activation函数传递它们。\n",
    "\n",
    "$$E_t = \\tanh(\\text{attn}(s_{t-1}, H))$$ \n",
    "\n",
    "这可以看作是计算每个编码器隐藏状态`匹配`前一个解码器隐藏状态的程度。\n",
    "\n",
    "我们当前对批次中每个`example`有一个 **[dec hid dim, src sent len]** 张量。 我们希望对批处理中的每个示例都使用 **[src sent len]** ，因为注意力应该超过源语句的长度。 这是通过将`energy`乘以 **[1, dec hid dim]** 张量, $v$。\n",
    "\n",
    "$$\\hat{a}_t = v E_t$$\n",
    "\n",
    "我们可以认为这对于编码器的每个隐藏层的状态，计算所有`dec_hid_dem`元素的“匹配”的加权和。 就如我们学习 $v$ 的参数一样，这里的权重也通过学习得。\n",
    "\n",
    "最后，通过传递 $\\text{softmax}$ 层，我们确保注意向量符合所有元素都在0和1之间、向量总和为1的约束条件。\n",
    "\n",
    "$$a_t = \\text{softmax}(\\hat{a_t})$$\n",
    "\n",
    "这让我们`attention`作用到源句!\n",
    "\n",
    "从图形上看，这看起来像下面这样。这是计算第一个注意向量, $s_{t-1} = s_0 = z$. 绿色/黄色块表示来自前向和后向RNNs的隐藏状态，注意计算都在粉色块中完成。\n",
    "\n",
    "![](assets/seq2seq9.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Attention(nn.Module):\n",
    "    def __init__(self, enc_hid_dim, dec_hid_dim):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.enc_hid_dim = enc_hid_dim\n",
    "        self.dec_hid_dim = dec_hid_dim\n",
    "        \n",
    "        self.attn = nn.Linear((enc_hid_dim * 2) + dec_hid_dim, dec_hid_dim)\n",
    "        self.v = nn.Parameter(torch.rand(dec_hid_dim))\n",
    "        \n",
    "    def forward(self, hidden, encoder_outputs):\n",
    "        \n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        #encoder_outputs = [src sent len, batch size, enc hid dim * 2]\n",
    "        \n",
    "        batch_size = encoder_outputs.shape[1]\n",
    "        src_len = encoder_outputs.shape[0]\n",
    "        \n",
    "        #repeat encoder hidden state src_len times\n",
    "        hidden = hidden.unsqueeze(1).repeat(1, src_len, 1)\n",
    "        \n",
    "        encoder_outputs = encoder_outputs.permute(1, 0, 2)\n",
    "        \n",
    "        #hidden = [batch size, src sent len, dec hid dim]\n",
    "        #encoder_outputs = [batch size, src sent len, enc hid dim * 2]\n",
    "        \n",
    "        energy = torch.tanh(self.attn(torch.cat((hidden, encoder_outputs), dim=2))) \n",
    "        \n",
    "        #energy = [batch size, src sent len, dec hid dim]\n",
    "        \n",
    "        energy = energy.permute(0, 2, 1)\n",
    "        \n",
    "        #energy = [batch size, dec hid dim, src sent len]\n",
    "        \n",
    "        #v = [dec hid dim]\n",
    "        \n",
    "        v = self.v.repeat(batch_size, 1).unsqueeze(1)\n",
    "        \n",
    "        #v = [batch size, 1, dec hid dim]\n",
    "                \n",
    "        attention = torch.bmm(v, energy).squeeze(1)\n",
    "        \n",
    "        #attention= [batch size, src len]\n",
    "        \n",
    "        return F.softmax(attention, dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoder\n",
    "\n",
    "\n",
    "解码器包含注意层, `attention`。注意层输入该层之前的隐藏状态, $s_{t-1}$, 和编码器所有的隐藏状态, $H$, 返回注意向量, $a_t$.\n",
    "\n",
    "然后，我们使用这个注意向量创建一个加权的源向量，$w_t$，用“加权”表示，它是编码器隐藏状态的加权和，$H$，使用 $a_t$ 作为权重。\n",
    "\n",
    "$$w_t = a_t H$$\n",
    "\n",
    "输入的单词 (已经被`embedded`), $y_t$, 加权源向量, $w_t$,以及先前解码器的隐藏状态, $s_{t-1}$, 然后全部传递到解码器RNN，将$y_t$和$w_t$连接在一起。\n",
    "\n",
    "$$s_t = \\text{DecoderGRU}(y_t, w_t, s_{t-1})$$\n",
    "\n",
    "然后，我们通过线性层$f$传递$y_t$、$w_t$和$s_t$，以预测目标句中的下一个单词$\\hat{y}_{t+1}$。这是通过将它们连接在一起来实现的。\n",
    "\n",
    "$$\\hat{y}_{t+1} = f(y_t, w_t, s_t)$$\n",
    "\n",
    "下图显示了一个示例翻译中的第一个单词的解码。\n",
    "\n",
    "![](assets/seq2seq10.png)\n",
    "\n",
    "绿色/黄色块显示向前/向后编码器RNN输出 $H$ ,红色方块显示上下文向量, $z = h_T = \\tanh(g(h^\\rightarrow_T,h^\\leftarrow_T)) = \\tanh(g(z^\\rightarrow, z^\\leftarrow)) = s_0$, 蓝色块表示解码器RNN输出$s_t$，紫色块表示线性层$f$，输出$\\hat{y}_{t+1}$，橙色块表示对$H$的加权和乘以$a_t$的计算，输出$w_t$。没有显示的是$a_t$的计算。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self, output_dim, emb_dim, enc_hid_dim, dec_hid_dim, dropout, attention):\n",
    "        super().__init__()\n",
    "\n",
    "        self.emb_dim = emb_dim\n",
    "        self.enc_hid_dim = enc_hid_dim\n",
    "        self.dec_hid_dim = dec_hid_dim\n",
    "        self.output_dim = output_dim\n",
    "        self.dropout = dropout\n",
    "        self.attention = attention\n",
    "        \n",
    "        self.embedding = nn.Embedding(output_dim, emb_dim)\n",
    "        \n",
    "        self.rnn = nn.GRU((enc_hid_dim * 2) + emb_dim, dec_hid_dim)\n",
    "        \n",
    "        self.out = nn.Linear((enc_hid_dim * 2) + dec_hid_dim + emb_dim, output_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, input, hidden, encoder_outputs):\n",
    "             \n",
    "        #input = [batch size]\n",
    "        #hidden = [batch size, dec hid dim]\n",
    "        #encoder_outputs = [src sent len, batch size, enc hid dim * 2]\n",
    "        \n",
    "        input = input.unsqueeze(0)\n",
    "        \n",
    "        #input = [1, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(input))\n",
    "        \n",
    "        #embedded = [1, batch size, emb dim]\n",
    "        \n",
    "        a = self.attention(hidden, encoder_outputs)\n",
    "                \n",
    "        #a = [batch size, src len]\n",
    "        \n",
    "        a = a.unsqueeze(1)\n",
    "        \n",
    "        #a = [batch size, 1, src len]\n",
    "        \n",
    "        encoder_outputs = encoder_outputs.permute(1, 0, 2)\n",
    "        \n",
    "        #encoder_outputs = [batch size, src sent len, enc hid dim * 2]\n",
    "        \n",
    "        weighted = torch.bmm(a, encoder_outputs)\n",
    "        \n",
    "        #weighted = [batch size, 1, enc hid dim * 2]\n",
    "        \n",
    "        weighted = weighted.permute(1, 0, 2)\n",
    "        \n",
    "        #weighted = [1, batch size, enc hid dim * 2]\n",
    "        \n",
    "        rnn_input = torch.cat((embedded, weighted), dim=2)\n",
    "        \n",
    "        #rnn_input = [1, batch size, (enc hid dim * 2) + emb dim]\n",
    "            \n",
    "        output, hidden = self.rnn(rnn_input, hidden.unsqueeze(0))\n",
    "        \n",
    "        #output = [sent len, batch size, dec hid dim * n directions]\n",
    "        #hidden = [n layers * n directions, batch size, dec hid dim]\n",
    "        \n",
    "        #sent len, n layers and n directions will always be 1 in this decoder, therefore:\n",
    "        #output = [1, batch size, dec hid dim]\n",
    "        #hidden = [1, batch size, dec hid dim]\n",
    "        #this also means that output == hidden\n",
    "        assert (output == hidden).all()\n",
    "        \n",
    "        embedded = embedded.squeeze(0)\n",
    "        output = output.squeeze(0)\n",
    "        weighted = weighted.squeeze(0)\n",
    "        \n",
    "        output = self.out(torch.cat((output, weighted, embedded), dim=1))\n",
    "        \n",
    "        #output = [bsz, output dim]\n",
    "        \n",
    "        return output, hidden.squeeze(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seq2Seq\n",
    "\n",
    "这是第一个模型，编码器RNN和解码器RNN不需要有相同的隐藏状态大小，但编码器必须是双向的。 \n",
    "\n",
    "这个seq2seq封装器与前两个类似。\n",
    "唯一的区别在于,编码器返回最后一个隐藏的状态作为(这是最后的隐藏状态的向前和向后编码器RNNs通过一个线性层得出的)解码器初始隐藏状态,以及每个隐藏状态(这是向前和向后的隐状态`stack`在一起)。我们还需要确保 `hidden`和`encoder_outputs`被传递给解码器。\n",
    "\n",
    "简单回顾一下所有的解码步骤:\n",
    "\n",
    "- `outputs`张量用于保存所有的预测，$\\hat{Y}$\n",
    "- 原序列, $X$, 输入编码器以得出 $z$ 和 $H$\n",
    "- 初始解码器隐藏状态设置为上下文向量, $s_0 = z = h_T$\n",
    "- 我们使用一批`<sos> `Token 作为第一个'input'，$y_1$\n",
    "- 然后我们在循环中解码:\n",
    "  - 将输入Token $y_t$、以前的隐藏状态$s_{t-1}$和所有编码器输出(`encoder_outputs`) $H$ 喂入解码器\n",
    "  - 接收一个预测 $\\hat{y}_{t+1}$ 和一个新的隐藏状态 $s_t$\n",
    "  - 然后，我们决定是否要`teacher force`，设置下一个适当的输入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder, device):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        self.device = device\n",
    "        \n",
    "    def forward(self, src, trg, teacher_forcing_ratio=0.5):\n",
    "        \n",
    "        #src = [src sent len, batch size]\n",
    "        #trg = [trg sent len, batch size]\n",
    "        #teacher_forcing_ratio is probability to use teacher forcing\n",
    "        #e.g. if teacher_forcing_ratio is 0.75 we use teacher forcing 75% of the time\n",
    "        \n",
    "        batch_size = src.shape[1]\n",
    "        max_len = trg.shape[0]\n",
    "        trg_vocab_size = self.decoder.output_dim\n",
    "        \n",
    "        #tensor to store decoder outputs\n",
    "        outputs = torch.zeros(max_len, batch_size, trg_vocab_size).to(self.device)\n",
    "        \n",
    "        #encoder_outputs is all hidden states of the input sequence, back and forwards\n",
    "        #hidden is the final forward and backward hidden states, passed through a linear layer\n",
    "        encoder_outputs, hidden = self.encoder(src)\n",
    "                \n",
    "        #first input to the decoder is the <sos> tokens\n",
    "        output = trg[0,:]\n",
    "        \n",
    "        for t in range(1, max_len):\n",
    "            output, hidden = self.decoder(output, hidden, encoder_outputs)\n",
    "            outputs[t] = output\n",
    "            teacher_force = random.random() < teacher_forcing_ratio\n",
    "            top1 = output.max(1)[1]\n",
    "            output = (trg[t] if teacher_force else top1)\n",
    "\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型训练 / 测试\n",
    "\n",
    "<hr>\n",
    "不再展开拉，请参照原教程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow-gpu",
   "language": "python",
   "name": "tensorflow-gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}