natural-language-processing/Learning-Phrase-Representations-using-RNN-Encoder–Decoder-for-Statistical-Machine-Translation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation\n",
    "\n",
    "Kyunghyun Cho,Kyunghyun Cho\n",
    "3 Sep 2014\n",
    "\n",
    "https://arxiv.org/pdf/1406.1078.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "1. RNN Encoder–Decoder，两个RNN网络分别作为encoder和decoder\n",
    "1. encoder将一个变长sequence映射到一个固定长度vector\n",
    "1. decoder将这个固定长度vector映射到一个变长sequence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "The proposed neural network architecture, which we will refer to as an RNN Encoder–Decoder, consists of two recurrent neural networks (RNN) that act as an encoder and a decoder pair. The encoder maps a variable-length source sequence to a fixed-length vector, and the decoder maps the vector representation back to a variable-length target sequence.\n",
    "\n",
    "## RNN Encoder–Decoder\n",
    "### Preliminary: Recurrent Neural Networks\n",
    "- a hidden state h\n",
    "- an optional output y which operates on a variable-length sequence x = (x1,...,xT).\n",
    "\n",
    "At each time step t, the hidden state h⟨t⟩ of the RNN is updated by\n",
    "\n",
    "$$h_{<t>}=f(h_{<t-1>},x_t)\\ (1)$$\n",
    "\n",
    "- f is a non-linear activation function\n",
    "\n",
    "The output at each timestep t is the conditional distribution p(xt | xt−1, . . . , x1)\n",
    "\n",
    "$$p(x_{t,j}=1|x_{x-1},\\cdots,x_1)=\\frac{exp(w_j h_{<t>})}{\\sum_{j'=1}^K exp(w_{j'} h_{<t>})}\\ (2)$$\n",
    "\n",
    "By combining these probabilities, we can compute the probability of the sequence x using\n",
    "\n",
    "$$p(x)=\\prod_{t=1}^T p(x_t|x_{t-1},\\cdots,x_1)\\ (3)$$\n",
    "\n",
    "### RNN Encoder–Decoder\n",
    "The encoder is an RNN that reads each symbol of an input sequence x sequentially. As it reads each symbol, the hidden state of the RNN changes according to Eq. (1). After reading the end of the sequence (marked by an end-of-sequence symbol), the hidden state of the RNN is a summary c of the whole input sequence.\n",
    "\n",
    "The decoder of the proposed model is another RNN which is trained to generate the output sequence by predicting the next symbol $y_t$ given the hidden state h⟨t⟩. However, unlike the RNN described in Sec. 2.1, both $y_t$ and h⟨t⟩ are also conditioned on $y_{t−1}$ and on the summary c of the input sequence. Hence, the hidden state of the decoder at time t is computed by,\n",
    "\n",
    "$$h_{<t>}=f(h_{<t-1>},y_{t-1},c)$$\n",
    "\n",
    "and similarly, the conditional distribution of the next symbol is\n",
    "\n",
    "$$P(y_t|y_{t-1},y_{t-2},\\cdots,y_1,c)=g(h_{<t>},y_{t-1},c)$$\n",
    "\n",
    "for given activation functions f and g (the latter must produce valid probabilities, e.g. with a softmax).\n",
    "\n",
    "![1](http://ou8qjsj0m.bkt.clouddn.com//17-8-8/16608032.jpg)\n",
    "\n",
    "The two components of the proposed RNN Encoder–Decoder are jointly trained to maximize the conditional log-likelihood\n",
    "\n",
    "$$max_{\\theta} \\frac{1}{N}\\sum_{n=1}^N log p_{\\theta}(y_n|x_n)\\ (4)$$\n",
    "\n",
    "where θ is the set of the model parameters and each (xn,yn) is an (input sequence, output sequence) pair from the training set.\n",
    "\n",
    "### Hidden Unit that Adaptively Remembers and Forgets\n",
    "![2](http://ou8qjsj0m.bkt.clouddn.com//17-8-8/60099754.jpg)\n",
    "\n",
    "Propose a new type of hidden unit (f in Eq. (1)) that has been motivated by the LSTM unit but is much simpler to compute and implement:\n",
    "\n",
    "First, the reset gate $r_j$ is computed by\n",
    "\n",
    "$$r_j=\\sigma([W_r x]_j + [U_r h_{t-1}]_j)\\ (5)$$\n",
    "\n",
    "where σ is the logistic sigmoid function, and $[.]_j$ denotes the j-th element of a vector. x and $h_{t-1}$ are the input and the previous hidden state, respectively. Wr and Ur are weight matrices which are learned.\n",
    "\n",
    "The update gate $z_j$ is computed by\n",
    "\n",
    "$$z_j=\\sigma([W_z x]_j + [U_z h_{<t-1>}]_j)\\ (6)$$\n",
    "\n",
    "The actual activation of the proposed unit $h_j$ is then computed by\n",
    "\n",
    "$$h_j^{<t>}=z_j h_j^{<t-1>}+(1-z_j) \\tilde{h}_j^{<t>}\\ (7)$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\\tilde{h}_j^{<t>}=\\phi([W_x]_j + [U(r \\odot h_{<t-1>})]_j)\\ (8)$$\n",
    "\n",
    "n this formulation, when the reset gate is close to 0, the hidden state is forced to ignore the previous hidden state and reset with the current input only. This effectively allows the hidden state to **drop** any information that is found to be irrelevant later in the future, thus, allowing a more compact representation.\n",
    "\n",
    "## Statistical Machine Translation\n",
    "The goal of the system (decoder, specifically) is to find a translation f given a source sentence e, which maximizes\n",
    "\n",
    "$$p(f|e) \\propto p(e|f)p(f)$$\n",
    "\n",
    "In practice, however, most SMT systems model log p(f | e) as a log- linear model with additional features and corresponding weights:\n",
    "\n",
    "$$log p(f|e)=\\sum_{n=1}^N w_n f_n(f,e)+log Z(e)\\ (9)$$\n",
    "\n",
    "where fn and wn are the n-th feature and weight, respectively. Z(e) is a normalization constant that does not depend on the weights. The weights are often optimized to maximize the BLEU score on a development set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}