natural-language-processing/Learning-Deep-Structured-Semantic-Models-for-Web-Search-using-Clickthrough-Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Deep Structured Semantic Models for Web Search using Clickthrough Data\n",
    "\n",
    "    Po-Sen Huang\n",
    "    Xiaodong He, Jianfeng Gao, Li Deng,Alex Acero, Larry Heck\n",
    "    CIKM’13, Oct. 27 2013\n",
    "\n",
    "https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/cikm2013_DSSM_fullversion.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "使用DNN将高维稀疏text features映射到低维稠密语义空间。\n",
    "\n",
    "- 输入层x是 bag-of-words term vectors(one-hot vector)\n",
    "- Word Hashing:reduce the dimensionality of the bag-of-words term vectors\n",
    "- activation function:tanh\n",
    "- 通过cos得到一个文档D和一个query Q的相似度R(Q,D)\n",
    "- 通过softmax得到基于给定Query最大的后验概率"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## INTRODUCTION\n",
    "These latent semantic models(`LSA`:latent semantic analysis) address the language discrepancy between Web documents and search queries by grouping different terms that occur in a similar context into the same semantic cluster. Thus, a query and a document, represented as two vectors in the lower-dimensional semantic space, can still have a high similarity score even if they do not share any term.\n",
    "\n",
    "`PLSA`/`LDA`:these models are often trained in an unsupervised manner using an objective function that is only loosely coupled with the evaluation metric for the retrieval task. Thus the performance of these models on Web search tasks is not as good as originally expected.\n",
    "\n",
    "Both `BLTM` and `DPM` outperform significantly the unsupervised latent semantic models, including LSA and PLSA, in the document ranking task.\n",
    "\n",
    "`deep auto-encoders`:They demonstrated that hierarchical semantic structure embedded in the query and the document can be extracted via deep learning.\n",
    "\n",
    "## RELATED WORK\n",
    "### Latent Semantic Models and the Use of Clickthrough Data\n",
    "- 通过SVD，一个document D可以映射成一个向量\n",
    "- 对Query做同样的事情\n",
    "- 相似度：这样就是比较两个向量的距离\n",
    "\n",
    "### Deep Learning\n",
    "They proposed a `semantic hashing` (SH) method which uses bottleneck features learned from the deep auto-encoder for information retrieval.\n",
    "\n",
    "1. a stack of generative models (i.e., the restricted Boltzmann machine) are learned to map layer-by-layer a term vector representation of a document to a low-dimensional semantic concept vector.\n",
    "1. the model parameters are fine-tuned so as to minimize the cross entropy error between the original term vector of the document and the reconstructed term vector.\n",
    "\n",
    "The intermediate layer activations are used as features (i.e., bottleneck) for document ranking.\n",
    "\n",
    "SH suffers from two problems:\n",
    "\n",
    "1. the model parameters are optimized for the re-construction of the document term vectors rather than for differentiating the relevant documents from the irrelevant ones for a given query. \n",
    "1. in order to make the computational cost manageable, the term vectors of documents consist of only the most-frequent 2000 words.\n",
    "\n",
    "## DEEP STRUCTURED SEMANTIC MODELS FOR WEB SEARCH\n",
    "### DNN for Computing Semantic Features\n",
    "![1](http://ou8qjsj0m.bkt.clouddn.com//17-8-6/33010786.jpg)\n",
    "\n",
    "- denote x as the input term vector\n",
    "- y as the output vector\n",
    "- $l_i, i=1,...,N-1$, as the intermediate hidden layers\n",
    "- $W_i$ as the i-th weight matrix\n",
    "- $b_i$ as the i-th bias term,\n",
    "\n",
    "$l_1=W_1x$\n",
    "\n",
    "$l_i=f(W_il_{i-1}+b_i),i=2,...,N-1$\n",
    "\n",
    "$y=f(W_Nl_{N-1}+b_N)$\n",
    "\n",
    "use the tanh as the activation function at the output layer and the hidden layers $l_i,i=2,...,N-1$:\n",
    "\n",
    "$f(x)=\\frac{1-e^{-2x}}{1+e^{-2x}}$\n",
    "\n",
    "The semantic relevance score between a query Q and a document D is then measured as:\n",
    "\n",
    "$R(Q,D)=cosine(y_Q,y_D)=\\frac{y_Q^T y_D}{\\Vert y_Q \\Vert\\Vert y_D \\Vert}$\n",
    "\n",
    "### WordHashing\n",
    "Given a word (e.g. good), we first add word starting and ending marks to the word (e.g. #good#). Then, we break the word into letter n-grams (e.g. letter trigrams: #go, goo, ood, od#). Finally, the word is represented using a vector of letter n-grams.\n",
    "\n",
    "Compared with the original size of the one-hot vector, word hashing allows us to represent a query or a document using a vector with much lower dimensionality.\n",
    "\n",
    "### Learning the DSSM\n",
    "we compute the posterior probability of a document given a query from the semantic relevance score between them through a softmax function\n",
    "\n",
    "$P(D|Q)=\\frac{exp(\\gamma R(Q,D))}{\\sum_{D_r \\in D} exp(\\gamma R(Q,D'))}$\n",
    "\n",
    "- $\\gamma$ is a smoothing factor in the softmax function\n",
    "- D denotes the set of candidate documents to be ranked\n",
    "\n",
    "In training, the model parameters are estimated to maximize\n",
    "the likelihood of the clicked documents given the queries across the training set. Equivalently, we need to minimize the following loss function\n",
    "\n",
    "$L(\\Lambda)=-log \\prod_{(Q,D^+)} P(D^+ | Q)$\n",
    "\n",
    "- $D^+$ is the clicked document\n",
    "- $\\Lambda$ denotes the papameter set of the neural networks {$W_i$,$b_i$}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}