examples/tutorials/faq_tutorial_tfidf_logreg.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FAQ task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FAQ consists of questions:\n",
    "1. What is preparatory course?\n",
    " - Preparatory course is a special educational program lasting 1 academic year (7-10 months), where students learn Russian and special disciplines (mathematics and physics).\n",
    "2. What is invitation letter?\n",
    " - The invitation is official document which is prepared by Ministry of Internal Affairs of Russian Federation. It confirms that the student is admitted to this university.\n",
    "3. ...\n",
    "\n",
    "\n",
    "Now you have questions from users and you need to answer, for example:\n",
    "\n",
    ":: Could I work while studying?\n",
    "> It allows the student to find well paid work and to start climbing up on a career ladder right after completing university course. Students of the Russian universities are obliged to attend all lectures as only the knowledge gained during classroom occupations allows students to become the effective and knowing professionals. \n",
    "\n",
    "\n",
    "In this tuorial we'll describe how to build FAQ model based on config deeppavlov/configs/faq/tfidf_logreg_en_faq.json\n",
    "<br>First of all we need train dataset of FAQ. As example, let's consider MIPT FAQ for entrants - https://mipt.ru/english/edu/faqs/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Please, install all necessary requirements using command:\n",
    "\n",
    ">\\>\\> python -m deeppavlov install deeppavlov/configs/faq/tfidf_logreg_en_faq.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the FAQ dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>Answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is preparatory course?</td>\n",
       "      <td>Preparatory course is a special educational pr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is invitation letter?</td>\n",
       "      <td>The invitation is official document which is p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is registration?</td>\n",
       "      <td>Registration grants to the foreign citizen the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Is it possible to study and work at the same t...</td>\n",
       "      <td>Russian education is one of the most qualitati...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>How long does the academic year last?</td>\n",
       "      <td>Academic year proceeds 10 months (from Septemb...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>What documents are demanded for admission?</td>\n",
       "      <td>Passport, documents of your previous education...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>What is the price for one year of study?</td>\n",
       "      <td>Russian taught programs cost 250'000 rubles pe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Should I insure my life?</td>\n",
       "      <td>Life insurance and health is obligatory for an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>In what cases student can be deducted from Uni...</td>\n",
       "      <td>At own will, for health reasons, for the acade...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>I have problems. Who can help me?</td>\n",
       "      <td>If you have any problems you can address to De...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            Question  \\\n",
       "0                        What is preparatory course?   \n",
       "1                         What is invitation letter?   \n",
       "2                              What is registration?   \n",
       "3  Is it possible to study and work at the same t...   \n",
       "4              How long does the academic year last?   \n",
       "5         What documents are demanded for admission?   \n",
       "6           What is the price for one year of study?   \n",
       "7                           Should I insure my life?   \n",
       "8  In what cases student can be deducted from Uni...   \n",
       "9                  I have problems. Who can help me?   \n",
       "\n",
       "                                              Answer  \n",
       "0  Preparatory course is a special educational pr...  \n",
       "1  The invitation is official document which is p...  \n",
       "2  Registration grants to the foreign citizen the...  \n",
       "3  Russian education is one of the most qualitati...  \n",
       "4  Academic year proceeds 10 months (from Septemb...  \n",
       "5  Passport, documents of your previous education...  \n",
       "6  Russian taught programs cost 250'000 rubles pe...  \n",
       "7  Life insurance and health is obligatory for an...  \n",
       "8  At own will, for health reasons, for the acade...  \n",
       "9  If you have any problems you can address to De...  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "FAQ_DATASET_URL = 'http://files.deeppavlov.ai/faq/mipt/faq.csv'\n",
    "faq_dataset = pd.read_csv(FAQ_DATASET_URL)\n",
    "faq_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import deeppavlov\n",
    "from deeppavlov.models.tokenizers.spacy_tokenizer import StreamSpacyTokenizer\n",
    "from deeppavlov.models.sklearn import SklearnComponent\n",
    "from deeppavlov.dataset_readers.faq_reader import FaqDatasetReader\n",
    "from deeppavlov.core.data.data_learning_iterator import DataLearningIterator\n",
    "from deeppavlov.core.data.utils import download_decompress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read FAQ data\n",
    "reader = FaqDatasetReader()\n",
    "faq_data = reader.read(data_url=FAQ_DATASET_URL, x_col_name='Question', y_col_name='Answer')\n",
    "iterator = DataLearningIterator(data=faq_data)\n",
    "\n",
    "x,y = iterator.get_instances()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train FAQ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's consider simple case for FAQ model (in the end you can find more complex pipeline models):\n",
    "1. TF_IDF vectorizer on lemmatized questions\n",
    "2. Logistic regression classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['academic',\n",
       " 'admission',\n",
       " 'and',\n",
       " 'at',\n",
       " 'be',\n",
       " 'can',\n",
       " 'case',\n",
       " 'course',\n",
       " 'deduct',\n",
       " 'demand',\n",
       " 'do',\n",
       " 'document',\n",
       " 'for',\n",
       " 'from',\n",
       " 'have',\n",
       " 'help',\n",
       " 'how',\n",
       " 'in',\n",
       " 'insure',\n",
       " 'invitation',\n",
       " 'last',\n",
       " 'letter',\n",
       " 'life',\n",
       " 'long',\n",
       " 'of',\n",
       " 'one',\n",
       " 'possible',\n",
       " 'preparatory',\n",
       " 'price',\n",
       " 'problem',\n",
       " 'registration',\n",
       " 'same',\n",
       " 'should',\n",
       " 'student',\n",
       " 'study',\n",
       " 'the',\n",
       " 'time',\n",
       " 'to',\n",
       " 'university',\n",
       " 'what',\n",
       " 'who',\n",
       " 'work',\n",
       " 'year']"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.model.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:30:09.280 WARNING in 'deeppavlov.core.models.serializable'['serializable'] at line 48: No load path is set for SklearnComponent in 'infer' mode. Using save path instead\n",
      "2019-02-12 12:30:09.280 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 218: Cannot load model from /home/vimary/ipavlov/Pilot/examples/tutorials/faq/tfidf.pkl\n",
      "2019-02-12 12:30:09.281 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 165: Initializing model sklearn.feature_extraction.text:TfidfVectorizer from scratch\n",
      "2019-02-12 12:30:09.281 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 108: Fitting model sklearn.feature_extraction.text:TfidfVectorizer\n",
      "2019-02-12 12:30:09.285 WARNING in 'deeppavlov.core.models.serializable'['serializable'] at line 48: No load path is set for SklearnComponent in 'infer' mode. Using save path instead\n",
      "2019-02-12 12:30:09.287 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 218: Cannot load model from /home/vimary/ipavlov/Pilot/examples/tutorials/faq/tfidf_logreg_classifier_en_mipt_faq.pkl\n",
      "2019-02-12 12:30:09.292 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 165: Initializing model sklearn.linear_model:LogisticRegression from scratch\n",
      "2019-02-12 12:30:09.294 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 108: Fitting model sklearn.linear_model:LogisticRegression\n"
     ]
    }
   ],
   "source": [
    "# create tokenizer\n",
    "tokenizer = StreamSpacyTokenizer(lemmas=True)\n",
    "x_tokenized = tokenizer(x)\n",
    "\n",
    "x_tokens_joined = tokenizer(x_tokenized)\n",
    "# fit TF-IDF vectorizer on train FAQ dataset \n",
    "vectorizer = SklearnComponent(model_class=\"sklearn.feature_extraction.text:TfidfVectorizer\",\n",
    "                              save_path='faq/tfidf.pkl',\n",
    "                              infer_method='transform')\n",
    "vectorizer.fit(x_tokens_joined)\n",
    "\n",
    "# Now collect (x,y) pairs: x_train - vectorized question, y_train - answer from FAQ\n",
    "x_train = vectorizer(x_tokens_joined)\n",
    "y_train = y \n",
    "\n",
    "# Let's use top 2 answers for each incoming questions (top_n param)\n",
    "clf = SklearnComponent(model_class=\"sklearn.linear_model:LogisticRegression\",\n",
    "                       top_n=2,\n",
    "                       c=1000,\n",
    "                       penalty='l2', \n",
    "                       save_path='faq/tfidf_logreg_classifier_en_mipt_faq.pkl',\n",
    "                       infer_method='predict')\n",
    "clf.fit(x_train, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test FAQ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_questions = ['Could you help me??', 'Could I work while studying?']\n",
    "tokenized_test_questions = tokenizer(test_questions)\n",
    "joined_test_q_tokens = tokenizer(tokenized_test_questions)\n",
    "test_q_vectorized = vectorizer(joined_test_q_tokens)\n",
    "answers = clf(test_q_vectorized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have all output of FAQ model: answers and scores.\n",
    "<br>\n",
    "Answers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answers 0:\n",
      "If you have any problems you can address to Department of Foreign Students: +7 (495) 408-70-43 (Auditorium building, room 315).\n",
      "\n",
      "Answers 1:\n",
      "Russian education is one of the most qualitative and fundamental in the world. It allows the student to find well paid work and to start climbing up on a career ladder right after completing university course. Students of the Russian universities are obliged to attend all lectures as only the knowledge gained during classroom occupations allows students to become the effective and knowing professionals. Thus, there is an opportunity to work only after classes or during vacation on the weekend.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, answer in enumerate(answers):\n",
    "    print('Answers {}:\\n{}\\n'.format(i, answer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Described model built in config - deeppavlov/configs/faq/tfidf_logreg_en_faq.json\n",
    "\n",
    "You can also combine different components to construct pipelines for FAQ task:\n",
    "\n",
    "Vectorizers:\n",
    " - deeppavlov.core.models.vectorizers.TfIdfVectorizer\n",
    " - deeppavlov.core.models.vectorizers.SentenceAvgW2vVectorizer\n",
    " - deeppavlov.core.models.vectorizers.SentenceW2vVectorizerTfidfWeights\n",
    "\n",
    "Classifiers:\n",
    " - deeppavlov.models.classifiers.logreg_classifier.LogregClassifier\n",
    " - deeppavlov.models.classifiers.cos_sim_classifier.CosineSimilarityClassifier\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}