natural-language-processing/Deep-Learning-for-Answer-Sentence-Selection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning for Answer Sentence Selection\n",
    "\n",
    "    Lei Yu,Karl Moritz Hermann,Phil Blunsom,Stephen Pulman\n",
    "    Department of Computer Science, University of Oxford\n",
    "    Google DeepMind\n",
    "    {lei.yu, phil.blunsom, stephen.pulman}@cs.ox.ac.uk \n",
    "    kmh@google.com\n",
    "\n",
    "https://arxiv.org/pdf/1412.1632.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "1. 可监督分类（q,a,y）,q-query，a-answer，y是1/0表示回答正确/错误\n",
    "1. query和answer都用word embedding表示（两种方式：Bag-of-words和Bigram CNN）\n",
    "1. 优化的目标函数是给定query优化到最大的答对数量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "In this paper, we show that a neural network-based sentence model can be applied to the task of **answer sentence selection**.\n",
    "\n",
    "## Model description\n",
    "Answer sentence selection can be viewed as a binary classification problem：\n",
    "\n",
    "- assume a set of questions Q\n",
    "- each question $q_i \\in Q$ is associated with a list of answer sentences $\\{a_{i1},a_{i2},\\cdots,a_{im}\\}$\n",
    "- together with their judgements $\\{y_{i1},y_{i2},\\cdots,y_{im}\\}$,where $y_{ij} =1$ if the answer is correct and $y_{ij} = 0$ otherwise\n",
    "\n",
    "While this could be approached as a multi-labelling task, we simply treat each data point as a triple $(q_i, a_{ij}, y_{ij})$. \n",
    "\n",
    "Our solution to this problem assumes that correct answers have high semantic similarity to questions.\n",
    "\n",
    "Formally, following Bordes et al., given the vector representations of a question q and an answer a (both in $\\mathbb{R^d}$), the probability of the answer being correct is\n",
    "\n",
    "$$p(y=1|q,a)=\\sigma(q^T M a + b),\\ (1)$$\n",
    "\n",
    "where the bias term b and the transformation matrix $M \\in \\mathbb{R^{d\\times d}}$ are model parameters.\n",
    "\n",
    "This formulation can intuitively be understood as an expression of the generative approach to open domain question answering: given a candidate answer sentence, we ‘generate’ a question through the transformation q′ = M a, and then measure the similarity of the generated question q′ and the given question q by their dot product. The sigmoid function squashes the similarity scores to a probability between 0 and 1.\n",
    "\n",
    "The model is trained by minimising the cross entropy of all labelled data QA pairs:\n",
    "\n",
    "$$L=-log\\prod_{n}p(y_n|q_n)+\\frac{\\lambda}{2}\\lVert\\theta\\rVert_{F}^2\\ (2)$$\n",
    "\n",
    "$$=-\\sum_{n}y_nlog\\sigma(q_n^TMa_n+b)+(1-y_n)log(1-\\sigma(q_n^TMa_n+b))+\\frac{\\lambda}{2}\\lVert\\theta\\rVert_{F}^2,$$\n",
    "\n",
    "where $\\lVert\\theta\\rVert_F^2$ is the Frobenius norm of θ,and θ includes {M,b} as well as any parameters introduced in the sentence composition model.\n",
    "\n",
    "### Bag-of-words model\n",
    "Given word embeddings, the bag-of-words model generates the vector representation of a sentence by summing over the embeddings of all words in the sentence—having previously removed stop words from the input. The vector is then normalised by the length of the sentence.\n",
    "\n",
    "$$s=\\frac{1}{|s|}\\sum_{i=1}^{|s|}s_i.\\ (3)$$\n",
    "\n",
    "### Bigram model\n",
    "The simple bag-of-words model proposed above is unable to capture more complex semantics of a sentence. To address this issue, we also evaluate a sentence model based on a convolutional neural network (CNN).\n",
    "\n",
    "![](http://ou8qjsj0m.bkt.clouddn.com//18-2-6/78462711.jpg)\n",
    "\n",
    "## Experiments\n",
    "![](http://ou8qjsj0m.bkt.clouddn.com//18-2-6/35837217.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}