natural-language-processing/Linguistic-Regularities-in-Continuous-Space-Word-Representations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linguistic Regularities in Continuous Space Word Representations\n",
    "\n",
    "    Tomas Mikolov\n",
    "    Wen-tau Yih\n",
    "    Geoffrey Zweig\n",
    "\n",
    "    NAACL-HLT 2013\n",
    "\n",
    "http://www.aclweb.org/anthology/N13-1090"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "这是一个RNN模型：\n",
    "\n",
    "$s(t)=f(Uw(t)+Ws(t-1))\\ (1)$\n",
    "\n",
    "$y(t)=g(Vs(t))\\ (2)$\n",
    "\n",
    "where\n",
    "\n",
    "$f(z)=\\frac{1}{1+e^{-z}},g(z_m)=\\frac{e^{z_m}}{\\sum_k e^{z_k}}\\ (3)$\n",
    "\n",
    "- w(t)是input vector(1-of-N coding,所以只有一个word)\n",
    "- U的每一列代表一个word（这是一个副产品，但是有很多用途）\n",
    "- y(t)是output vector，是一个词的概率分布模型\n",
    "- s(t)是当前的句子的vector表示\n",
    "- s(t-1)是上一句句子的vector表示\n",
    "- f是sigmoid函数\n",
    "- g是softmax函数\n",
    "\n",
    "提到了两种应用场景，都是\"if a:b, then c:d\",给定a,b,c，求d。\n",
    "\n",
    "- 场景1（POS）：good:better rough:\n",
    "- 场景2（语义）clothing is to shirt as dish is to bowl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent Neural Network Model\n",
    "![1](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/4426413.jpg)\n",
    "\n",
    "1. The input vector w(t) represents input word at time t encoded using 1-of-N coding（one hot encoding）, \n",
    "1. the output layer y(t) produces a probability distribution over words. \n",
    "1. The hidden layer s(t) maintains a representation of the sentence history. \n",
    "1. The input vector w(t) and the output vector y(t) have dimensionality of the vocabulary. \n",
    "1. The values in the hidden and output layers are computed as follows:\n",
    "\n",
    "$s(t)=f(Uw(t)+Ws(t-1))\\ (1)$\n",
    "\n",
    "$y(t)=g(Vs(t))\\ (2)$\n",
    "\n",
    "where\n",
    "\n",
    "$f(z)=\\frac{1}{1+e^{-z}},g(z_m)=\\frac{e^{z_m}}{\\sum_k e^{z_k}}\\ (3)$\n",
    "\n",
    "In this framework, the word representations are found in the columns of U, with each column representing a word.\n",
    "\n",
    "## Measuring Linguistic Regularity\n",
    "### A Syntactic Test Set\n",
    "Category | Relation | Patterns Tested | # Questions | Example\n",
    "---------|----------|-----------------|-------------|------\n",
    "Adjectives | Base/Comparative | JJ/JJR, JJR/JJ | 1000 | good:better rough:_ _\n",
    "Adjectives | Base/Superlative | JJ/JJS, JJS/JJ | 1000 | good:best rough:_ _\n",
    "Adjectives | Comparative/ Superlative | JJS/JJR, JJR/JJS | 1000 | better:best rougher:_ _\n",
    "Nouns | Singular/Plural | NN/NNS, NNS/NN | 1000 | year:years law:_ _\n",
    "Nouns | Non-possessive/ Possessive | NN/NN POS, NN POS/NN | 1000 | city:city’s bank:_ _\n",
    "Verbs | Base/Past | VB/VBD, VBD/VB | 1000 | see:saw return:_ _\n",
    "Verbs | Base/3rd Person Singular Present | VB/VBZ, VBZ/VB | 1000 | see:sees return:_ _\n",
    "Verbs | Past/3rd Person Singular Present | VBD/VBZ, VBZ/VBD | 1000 | saw:sees returned:_ _\n",
    "\n",
    "Table 1: Test set patterns. For a given pattern and word-pair, both orderings occur in the test set. For example, if “see:saw return:_ _ _” occurs, so will “saw:see returned:_ _ _ ”.\n",
    "\n",
    "### A Semantic Test Set\n",
    "To measure the degree that a target word pair dish:bowl has the same relation, we form the analogy “clothing is to shirt as dish is to bowl,” and ask how valid it is.\n",
    "\n",
    "## The Vector Offset Method\n",
    "In this model, to answer the analogy question a:b c:d where d is unknown, we find the embedding vectors $x_a,x_b,x_c$ (all normalized to unit norm), and compute $y=x_b-x_a+x_c$. y is the continuous space representation of the word we expect to be the best answer. \n",
    "\n",
    "We then search for the word whose embedding vector has the greatest cosine similarity to y and output it: $w^*=argmax_w\\frac{x_wy}{\\lVert x_w \\rVert \\lVert y \\rVert}$\n",
    "\n",
    "When d is given, as in our semantic test set, we simply use $cos(x_b-x_a+x_c,x_d)$ for the words provided.\n",
    "\n",
    "![3](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/19451168.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}