natural-language-processing/Learning-Continuous-Phrase-Representations-for-Translation-Modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Continuous Phrase Representations for Translation Modeling\n",
    "\n",
    "    Jianfeng Gao,Xiaodong He,Wen-tau Yih,Li Deng\n",
    "    2013\n",
    "\n",
    "https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/nn4smt.acl_.v9.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "1. 训练集：句子pair（源，目标），即（f，e）\n",
    "1. 无论是source还是target\n",
    "    1. 把bag-of-words representation\u001b转成 low-dimensional continuous space\n",
    "    1. 再求（f，e）相似度\n",
    "    1. 用softmax计算𝑃(𝐸|𝐹𝑖) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "The phrase translation model, also known as the phrase table, is one of the core components of phrase-based `statistical machine translation` (`SMT`) systems.The most common method of constructing the phrase table takes a two-phase approach.\n",
    "\n",
    "- First, the bilingual phrase pairs are extracted heuristically from an automatically word-aligned training data.\n",
    "- The second phase, which is the focus of this paper, is parameter estimation where each phrase pair is assigned with some scores that are estimated based on counting these phrases or their words using the same word-aligned training data.\n",
    "\n",
    "This paper revisits the problem of scoring a phrase translation pair by developing a `Continuous-space Phrase Translation Model` (`CPTM`).\n",
    "\n",
    "- First, we represent each phrase as a bag-of-words vector, called `word vector` henceforth.\n",
    "- We then project the word vector, in either the source language or the target language, into a respective continuous feature vector in a common low-dimensional space that is language independent. The projection is performed by a multi-layer neural network. The projected feature vector forms the `continuous representation` of a phrase.\n",
    "- Finally, the translation score of a source-target phrase pair is computed by the distance between their feature vectors.\n",
    "\n",
    "## The Log-Linear Model for SMT\n",
    "Phrase-based SMT is based on a log-linear model which requires learning a mapping between input 𝐹 ∈ ℱ to output 𝐸 ∈ ℰ. We are given\n",
    "\n",
    "- Training samples (𝐹𝑖, 𝐸𝑖) for 𝑖 = 1 … 𝑁, where each source sentence 𝐹𝑖 is paired with a reference translation in target language 𝐸𝑖;\n",
    "- A procedure GEN to generate a list of N-best candidates GEN(𝐹𝑖) for an input 𝐹𝑖, each 𝐸 ∈ GEN(𝐹𝑖) is labeled by the sentence-level BLEU score (He and Deng 2012), denoted by sBleu(𝐸𝑖,𝐸) , which measures the quality of 𝐸 with respect to its reference translation 𝐸𝑖;\n",
    "- A vector of features $h \\in \\mathbb{R}^M$ that maps each(𝐹𝑖, 𝐸) to a vector of feature values;\n",
    "- A parameter vector $\\lambda \\in \\mathbb{R}^M$, which assigns a real-valued weight to each feature. \n",
    "\n",
    "$$E^*=argmax_{(E,A) \\in GEN(F_i)} \\lambda^Th(F_i,E,A)\\ (1)$$\n",
    "\n",
    "which states that given 𝛌 and 𝐡, argmax returns the highest scoring translation $E^*$, maximizing over correspondences 𝐴.\n",
    "\n",
    "## A Continuous-Space Phrase Translation Model (CPTM)\n",
    "We start with a bag-of-words representation of a phrase $x \\in \\mathbb{R}^d$, where 𝐱 is a word vector and 𝑑 is the size of the vocabulary consisting of words in both source and target languages, which is set to 200K in our experiments. We then learn to project\n",
    "𝐱 to a low-dimensional continuous space $\\mathbb{R}^k$:\n",
    "\n",
    "$$\\phi(x):\\mathbb{R}^d \\to \\mathbb{R}^k$$\n",
    "\n",
    "Let 𝐖1 be the projection matrix from the input layer to the hidden layer and 𝐖2 the projection matrix from the hidden layer to the output layer, we have\n",
    "\n",
    "$$y \\equiv \\phi(x) = tanh(W_2^T(tanh(W_1^T x)))\\ (2)$$\n",
    "\n",
    "$$score(f,e) \\equiv sim_{\\theta}(x_f,x_e)=y_f^T y_e\\ (3)$$\n",
    "\n",
    "![1](http://ou8qjsj0m.bkt.clouddn.com//17-8-8/90077512.jpg)\n",
    "\n",
    "![2](http://ou8qjsj0m.bkt.clouddn.com//17-8-8/7292310.jpg)\n",
    "\n",
    "The CPTM of (2) and (3) can be incorporated into the log-linear model for SMT (1) by introducing a new feature ℎ𝑀+1 and a new feature weight 𝜆𝑀+1. The new feature is defined as\n",
    "\n",
    "$$h_{M+1}(F_i,E,A)=\\sum_{(f,e) \\in A} sim_{\\theta}(x_f,x_e)\\ (4)$$\n",
    "\n",
    "## Training CPTM\n",
    "We define the loss function ℒ(𝛉) as the negative of the N-best list based expected BLEU, denoted by xBleu(𝛉)\n",
    "\n",
    "$$xBleu(\\theta)=\\sum_{E \\in GEN(F_i)} P(E|F_i)sBleu(E_i,E)\\ (5)$$\n",
    "\n",
    "where sBleu(𝐸𝑖, 𝐸) is the sentence-level BLEU score, and 𝑃(𝐸|𝐹𝑖) is the translation probability from 𝐹𝑖 to 𝐸 computed using softmax as \n",
    "\n",
    "$$P(E|F_i)=\\frac{exp(\\gamma\\lambda^T h(F_i,E,A))}{\\sum_{E' \\in GEN(F_i)}exp(\\gamma\\lambda^T h(F_i,E',A))}\\ (6)$$\n",
    "\n",
    "where $\\lambda^T h$ is the log-linear model of (1), which also includes the feature derived from the CPTM as defined by (4), and 𝛾 is a tuned smoothing factor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}