examples/tutorials/05_deeppavlov_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You can also run the notebook in [COLAB](https://colab.research.google.com/github/deepmipt/DeepPavlov/blob/master/examples/tutorials/05_deeppavlov_classification.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install deeppavlov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification on DeepPavlov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task**:\n",
    "Intent recognition on SNIPS dataset: https://github.com/snipsco/nlu-benchmark/tree/master/2017-06-custom-intent-engines that has already been recomposed to `csv` format and can be downloaded from http://files.deeppavlov.ai/datasets/snips_intents/train.csv\n",
    "\n",
    "FastText English word embeddings ~8Gb: http://files.deeppavlov.ai/deeppavlov_data/embeddings/wiki.en.bin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plan of the notebook with documentation links:\n",
    "\n",
    "1. [Data aggregation](#Data-aggregation)\n",
    "     * [DatasetReader](#DatasetReader): [docs link](https://deeppavlov.readthedocs.io/en/latest/apiref/dataset_readers.html)\n",
    "     * [DatasetIterator](#DatasetIterator): [docs link](https://deeppavlov.readthedocs.io/en/latest/apiref/dataset_iterators.html)\n",
    "2. [Data preprocessing](#Data-preprocessing): [docs link](https://deeppavlov.readthedocs.io/en/latest/components/data_processors.html)\n",
    "     * [Lowercasing](#Lowercasing)\n",
    "     * [Tokenization](#Tokenization)\n",
    "     * [Vocabulary](#Vocabulary)\n",
    "3. [Featurization](#Featurization): [docs link](https://deeppavlov.readthedocs.io/en/latest/components/data_processors.html), [pre-trained embeddings link](https://deeppavlov.readthedocs.io/en/latest/intro/pretrained_vectors.html)\n",
    "    * [Bag-of-words embedder](#Bag-of-words)\n",
    "    * [TF-IDF vectorizer](#TF-IDF Vectorizer)\n",
    "    * [GloVe embedder](#GloVe-embedder)\n",
    "    * [Mean GloVe embedder](#Mean-GloVe-embedder)\n",
    "    * [GloVe weighted by TF-IDF embedder](#GloVe-weighted-by-TF-IDF-embedder)\n",
    "4. [Models](#Models): [docs link](https://deeppavlov.readthedocs.io/en/latest/components/classifiers.html)\n",
    "    * [Building models in python](#Models-in-python)\n",
    "        - [Sklearn component classifiers](#SklearnComponent-classifier-on-Tfidf-features-in-python)\n",
    "        - [Keras classification model on GloVe emb](#KerasClassificationModel-on-GloVe-embeddings-in-python)\n",
    "        - [Sklearn component classifier on GloVe weighted emb](#SklearnComponent-classifier-on-GloVe-weighted-by-TF-IDF-embeddings-in-python)\n",
    "    * [Building models from configs](#Models-from-configs)\n",
    "        - [Sklearn component classifiers](#SklearnComponent-classifier-on-Tfidf-features-from-config)\n",
    "        - [Keras classification model](#KerasClassificationModel-on-fastText-embeddings-from-config)\n",
    "        - [Sklearn component classifier on GloVe weighted emb](#SklearnComponent-classifier-on-GloVe-weighted-by-TF-IDF-embeddings-from-config)\n",
    "    * [Bonus: pre-trained CNN model in DeepPavlov](#Bonus:-pre-trained-CNN-model-in-DeepPavlov)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data aggregation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, let's download and look into data we will work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:21.101 INFO in 'deeppavlov.core.data.utils'['utils'] at line 63: Downloading from http://files.deeppavlov.ai/datasets/snips_intents/train.csv to snips/train.csv\n",
      "100%|██████████| 981k/981k [00:00<00:00, 63.5MB/s]\n"
     ]
    }
   ],
   "source": [
    "from deeppavlov.core.data.utils import simple_download\n",
    "\n",
    "#download train data file for SNIPS\n",
    "simple_download(url=\"http://files.deeppavlov.ai/datasets/snips_intents/train.csv\", \n",
    "                destination=\"./snips/train.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text,intents\r\n",
      "Add another song to the Cita RomГЎntica playlist. ,AddToPlaylist\r\n",
      "add clem burke in my playlist Pre-Party R&B Jams,AddToPlaylist\r\n",
      "Add Live from Aragon Ballroom to Trapeo,AddToPlaylist\r\n",
      "add Unite and Win to my night out,AddToPlaylist\r\n",
      "Add track to my Digster Future Hits,AddToPlaylist\r\n",
      "add the piano bar to my Cindy Wilson,AddToPlaylist\r\n",
      "Add Spanish Harlem Incident to cleaning the house,AddToPlaylist\r\n",
      "add The Greyest of Blue Skies in Indie EspaГ±ol my playlist,AddToPlaylist\r\n",
      "Add the name kids in the street to the plylist New Indie Mix,AddToPlaylist\r\n",
      "add album radar latino,AddToPlaylist\r\n",
      "Add Tranquility to the Latin Pop Rising playlist. ,AddToPlaylist\r\n",
      "Add d flame to the Dcode2016 playlist.,AddToPlaylist\r\n",
      "Add album to my fairy tales,AddToPlaylist\r\n",
      "I need another artist in the New Indie Mix playlist. ,AddToPlaylist\r\n"
     ]
    }
   ],
   "source": [
    "! head -n 15 snips/train.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DatasetReader\n",
    "\n",
    "Read data using `BasicClassificationDatasetReader` из DeepPavlov"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from deeppavlov.dataset_readers.basic_classification_reader import BasicClassificationDatasetReader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:23.376 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 96: Cannot find snips/valid.csv file\n",
      "2019-02-12 12:14:23.376 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 96: Cannot find snips/test.csv file\n"
     ]
    }
   ],
   "source": [
    "# read data from particular columns of `.csv` file\n",
    "dr = BasicClassificationDatasetReader().read(\n",
    "    data_path='./snips/',\n",
    "    train='train.csv',\n",
    "    x = 'text',\n",
    "    y = 'intents'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't have a ready train/valid/test split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('train', 15884), ('valid', 0), ('test', 0)]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check train/valid/test sizes\n",
    "[(k, len(dr[k])) for k in dr.keys()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DatasetIterator\n",
    "\n",
    "Use `BasicClassificationDatasetIterator` to split `train` on `train` and `valid` and to generate batches of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov.dataset_iterators.basic_classification_iterator import BasicClassificationDatasetIterator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:23.557 INFO in 'deeppavlov.dataset_iterators.basic_classification_iterator'['basic_classification_iterator'] at line 73: Splitting field <<train>> to new fields <<['train', 'valid']>>\n"
     ]
    }
   ],
   "source": [
    "# initialize data iterator splitting `train` field to `train` and `valid` in proportion 0.8/0.2\n",
    "train_iterator = BasicClassificationDatasetIterator(\n",
    "    data=dr,\n",
    "    field_to_split='train',  # field that will be splitted\n",
    "    split_fields=['train', 'valid'],   # fields to which the fiald above will be splitted\n",
    "    split_proportions=[0.8, 0.2],  #proportions for splitting\n",
    "    split_seed=23,  # seed for splitting dataset\n",
    "    seed=42)  # seed for iteration over dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look into training samples. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: Is it freezing in Offerman, California?\n",
      "y: ['GetWeather']\n",
      "=================\n",
      "x: put this song in the playlist Trap Land\n",
      "y: ['AddToPlaylist']\n",
      "=================\n",
      "x: show me a textbook with a rating of 2 and a maximum rating of 6 that is current\n",
      "y: ['RateBook']\n",
      "=================\n",
      "x: Will the weather be okay in Northern Luzon Heroes Hill National Park 4 and a half months from now?\n",
      "y: ['GetWeather']\n",
      "=================\n",
      "x: Rate the current album a four\n",
      "y: ['RateBook']\n",
      "=================\n"
     ]
    }
   ],
   "source": [
    "# one can get train instances (or any other data type including `all`)\n",
    "x_train, y_train = train_iterator.get_instances(data_type='train')\n",
    "for x, y in list(zip(x_train, y_train))[:5]:\n",
    "    print('x:', x)\n",
    "    print('y:', y)\n",
    "    print('=================')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using lowercasing and tokenization as data preparation. \n",
    "\n",
    "DeepPavlov also contains several other preprocessors and tokenizers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lowercasing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`StrLower` lowercases texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/vimary/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to /home/vimary/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package perluniprops to\n",
      "[nltk_data]     /home/vimary/nltk_data...\n",
      "[nltk_data]   Package perluniprops is already up-to-date!\n",
      "[nltk_data] Downloading package nonbreaking_prefixes to\n",
      "[nltk_data]     /home/vimary/nltk_data...\n",
      "[nltk_data]   Package nonbreaking_prefixes is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "from deeppavlov.models.preprocessors.str_lower import StrLower"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['is it freezing in offerman, california?']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str_lower = StrLower()\n",
    "str_lower(['Is it freezing in Offerman, California?'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization\n",
    "\n",
    "`NLTKTokenizer` can split string to tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov.models.tokenizers.nltk_moses_tokenizer import NLTKMosesTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Is', 'it', 'freezing', 'in', 'Offerman', ',', 'California', '?']]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = NLTKMosesTokenizer()\n",
    "tokenizer(['Is it freezing in Offerman, California?'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's preprocess all `train` part of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x_lower_tokenized = str_lower(tokenizer(train_iterator.get_instances(data_type='train')[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vocabulary\n",
    "\n",
    "Now we are ready to use `vocab`. They are very usefull for:\n",
    "* extracting class labels and converting labels to indices and vice versa,\n",
    "* building of characters or tokens vocabularies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov.core.data.simple_vocab import SimpleVocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize simple vocabulary to collect all appeared in the dataset classes\n",
    "classes_vocab = SimpleVocabulary(\n",
    "    save_path='./snips/classes.dict',\n",
    "    load_path='./snips/classes.dict')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:25.35 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 89: [saving vocabulary to /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n"
     ]
    }
   ],
   "source": [
    "classes_vocab.fit((train_iterator.get_instances(data_type='train')[1]))\n",
    "classes_vocab.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what classes the dataset contains and their indices in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('GetWeather', 0),\n",
       " ('PlayMusic', 1),\n",
       " ('SearchScreeningEvent', 2),\n",
       " ('BookRestaurant', 3),\n",
       " ('RateBook', 4),\n",
       " ('SearchCreativeWork', 5),\n",
       " ('AddToPlaylist', 6)]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(classes_vocab.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# also one can collect vocabulary of textual tokens appeared 2 and more times in the dataset\n",
    "token_vocab = SimpleVocabulary(\n",
    "    save_path='./snips/tokens.dict',\n",
    "    load_path='./snips/tokens.dict',\n",
    "    min_freq=2,\n",
    "    special_tokens=('<PAD>', '<UNK>',),\n",
    "    unk_token='<UNK>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:25.157 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 89: [saving vocabulary to /home/vimary/ipavlov/Pilot/examples/tutorials/snips/tokens.dict]\n"
     ]
    }
   ],
   "source": [
    "token_vocab.fit(train_x_lower_tokenized)\n",
    "token_vocab.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4564"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of tokens in dictionary\n",
    "len(token_vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 6953),\n",
       " ('a', 3917),\n",
       " ('in', 3265),\n",
       " ('to', 3203),\n",
       " ('for', 2814),\n",
       " ('of', 2401),\n",
       " ('.', 2400),\n",
       " ('i', 2079),\n",
       " ('at', 1935),\n",
       " ('play', 1703)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 10 most common words and number of times their appeared\n",
    "token_vocab.freqs.most_common()[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[13, 36, 244, 4, 1, 29, 996, 20]]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_ids = token_vocab(str_lower(tokenizer(['Is it freezing in Offerman, California?'])))\n",
    "token_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['is it freezing in <UNK>, california?']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(token_vocab(token_ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Featurization\n",
    "\n",
    "This part contains several possible ways of featurization of text samples. One can chose any appropriate vectorizer/embedder according to available resources and given task.\n",
    "\n",
    "Bag-of-words (BoW) and TF-IDF vectorizers converts text samples to vectors (one vector per sample) while fastText, GloVe, fastText weighted by TF-IDF embedders either produce an embedding vector per token or an embedding vector per text sample (if `mean` set to True)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bag-of-words\n",
    "\n",
    "Matches a vector to each text sample indicating which words appeared in the given sample: text -> binary vector $v$: \\[0, 1, 0, 0, 0, 1, ..., ...1, 0, 1\\]. \n",
    "\n",
    "Dimensionality of vector $v$ is equal to vocabulary size.\n",
    "\n",
    "$v_i$ == 1, if word $i$ is in the text,\n",
    "\n",
    "$v_i$ == 0, else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from deeppavlov.models.embedders.bow_embedder import BoWEmbedder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([0, 1, 0, ..., 0, 0, 0], dtype=int32)]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# initialize bag-of-words embedder giving total number of tokens\n",
    "bow = BoWEmbedder(depth=token_vocab.len)\n",
    "# it assumes indexed tokenized samples\n",
    "bow(token_vocab(str_lower(tokenizer(['Is it freezing in Offerman, California?']))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# all 8 tokens are in the vocabulary\n",
    "sum(bow(token_vocab(str_lower(tokenizer(['Is it freezing in Offerman, California?']))))[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-IDF Vectorizer\n",
    "\n",
    "Matches a vector to each text sample: text -> vector $v$ from $R^N$ where $N$ is a vocabulary size.\n",
    "\n",
    "$TF-IDF(token, document) = TF(token, document) * IDF(token, document)$\n",
    "\n",
    "$TF$ is a term frequency:\n",
    "\n",
    "$TF(token, document) = \\frac{n_{token}}{\\sum_{k}n_k}.$\n",
    "\n",
    "$IDF$ is a inverse document frequency:\n",
    "\n",
    "$IDF(token, all\\_documents) = \\frac{Total\\ number\\ of\\ documents}{number\\ of\\ documents\\ where\\ token\\ appeared}.$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`SklearnComponent` in DeepPavlov is a universal wrapper for any vecotirzer/estimator from `sklearn` package. The only requirement to specify component usage is following: model class and name of infer method should be passed as parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov.models.sklearn import SklearnComponent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:25.268 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 218: Cannot load model from /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v0.pkl\n",
      "2019-02-12 12:14:25.269 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 165: Initializing model sklearn.feature_extraction.text:TfidfVectorizer from scratch\n"
     ]
    }
   ],
   "source": [
    "# initialize TF-IDF vectorizer sklearn component with `transform` as infer method\n",
    "tfidf = SklearnComponent(\n",
    "    model_class=\"sklearn.feature_extraction.text:TfidfVectorizer\",\n",
    "    infer_method=\"transform\",\n",
    "    save_path='./tfidf_v0.pkl',\n",
    "    load_path='./tfidf_v0.pkl',\n",
    "    mode='train')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:25.296 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 108: Fitting model sklearn.feature_extraction.text:TfidfVectorizer\n",
      "2019-02-12 12:14:25.395 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 240: Saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v0.pkl\n"
     ]
    }
   ],
   "source": [
    "# fit on textual train instances and save it\n",
    "tfidf.fit(str_lower(train_iterator.get_instances(data_type='train')[0]))\n",
    "tfidf.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x10709 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 6 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf(str_lower(['Is it freezing in Offerman, California?']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10709"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of tokens in the TF-IDF vocabulary\n",
    "len(tfidf.model.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GloVe embedder\n",
    "\n",
    "[GloVe](https://nlp.stanford.edu/projects/glove/) is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from deeppavlov.models.embedders.glove_embedder import GloVeEmbedder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's download GloVe embedding file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:26.153 INFO in 'deeppavlov.core.data.utils'['utils'] at line 63: Downloading from http://files.deeppavlov.ai/embeddings/glove.6B.100d.txt to glove.6B.100d.txt\n",
      "347MB [00:06, 50.0MB/s] \n"
     ]
    }
   ],
   "source": [
    "simple_download(url=\"http://files.deeppavlov.ai/embeddings/glove.6B.100d.txt\", \n",
    "                destination=\"./glove.6B.100d.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:33.99 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 52: [loading GloVe embeddings from `/home/vimary/ipavlov/Pilot/examples/tutorials/glove.6B.100d.txt`]\n"
     ]
    }
   ],
   "source": [
    "embedder = GloVeEmbedder(load_path='./glove.6B.100d.txt',\n",
    "                         dim=100, pad_zero=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 8, (100,))"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# output shape is (batch_size x max_num_tokens_in_the_batch x embedding_dim)\n",
    "embedded_batch = embedder(str_lower(tokenizer(['Is it freezing in Offerman, California?']))) \n",
    "len(embedded_batch), len(embedded_batch[0]), embedded_batch[0][0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mean GloVe embedder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Embedder returns a vector per token while we want to get a vector per text sample. Therefore, let's calculate mean vector of embeddings of tokens. \n",
    "For that we can either init `GloVeEmbedder` with `mean=True` parameter (`mean=false` by default), or pass `mean=true` while calling function (this way `mean` value is assigned only for this call)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, (100,))"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# output shape is (batch_size x embedding_dim)\n",
    "embedded_batch = embedder(str_lower(tokenizer(['Is it freezing in Offerman, California?'])), mean=True) \n",
    "len(embedded_batch), embedded_batch[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GloVe weighted by TF-IDF embedder\n",
    "\n",
    "One of the possible ways to combine TF-IDF vectorizer and any token embedder is to weigh token embeddings by TF-IDF coefficients (therefore, `mean` set to True is obligatory to obtain embeddings of interest while it still **by default** returns embeddings of tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov.models.embedders.tfidf_weighted_embedder import TfidfWeightedEmbedder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "weighted_embedder = TfidfWeightedEmbedder(\n",
    "    embedder=embedder,  # our GloVe embedder instance\n",
    "    tokenizer=tokenizer,  # our tokenizer instance\n",
    "    mean=True,  # to return one vector per sample\n",
    "    vectorizer=tfidf  # our TF-IDF vectorizer\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, (100,))"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# output shape is (batch_size x  embedding_dim)\n",
    "embedded_batch = weighted_embedder(str_lower(tokenizer(['Is it freezing in Offerman, California?']))) \n",
    "len(embedded_batch), embedded_batch[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov.metrics.accuracy import sets_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all train and valid data from iterator\n",
    "x_train, y_train = train_iterator.get_instances(data_type=\"train\")\n",
    "x_valid, y_valid = train_iterator.get_instances(data_type=\"valid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models in python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SklearnComponent classifier on Tfidf-features in python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:53.75 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 218: Cannot load model from /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v0.pkl\n",
      "2019-02-12 12:14:53.75 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 165: Initializing model sklearn.linear_model:LogisticRegression from scratch\n"
     ]
    }
   ],
   "source": [
    "# initialize sklearn classifier, all parameters for classifier could be passed\n",
    "cls = SklearnComponent(\n",
    "    model_class=\"sklearn.linear_model:LogisticRegression\",\n",
    "    infer_method=\"predict\",\n",
    "    save_path='./logreg_v0.pkl',\n",
    "    load_path='./logreg_v0.pkl',\n",
    "    C=1,\n",
    "    mode='train')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:53.591 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 108: Fitting model sklearn.linear_model:LogisticRegression\n",
      "2019-02-12 12:14:53.756 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 240: Saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v0.pkl\n"
     ]
    }
   ],
   "source": [
    "# fit sklearn classifier and save it\n",
    "cls.fit(tfidf(x_train), y_train)\n",
    "cls.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_valid_pred = cls(tfidf(x_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text sample: I need seating at Floating restaurant in Tennessee for a group of 9\n",
      "True label: ['BookRestaurant']\n",
      "Predicted label: BookRestaurant\n"
     ]
    }
   ],
   "source": [
    "# Let's look into obtained result\n",
    "print(\"Text sample: {}\".format(x_valid[0]))\n",
    "print(\"True label: {}\".format(y_valid[0]))\n",
    "print(\"Predicted label: {}\".format(y_valid_pred[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.982373308152345"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's calculate sets accuracy (because each element is a list of labels)\n",
    "sets_accuracy(np.squeeze(y_valid), y_valid_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### KerasClassificationModel on GloVe embeddings in python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov.models.classifiers.keras_classification_model import KerasClassificationModel\n",
    "from deeppavlov.models.preprocessors.one_hotter import OneHotter\n",
    "from deeppavlov.models.classifiers.proba2labels import Proba2Labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:14:54.421 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 272: [initializing `KerasClassificationModel` from scratch as cnn_model]\n",
      "2019-02-12 12:14:54.818 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 136: Model was successfully initialized!\n",
      "Model summary:\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, 15, 100)      0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, 15, 128)      38528       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_2 (Conv1D)               (None, 15, 128)      64128       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, 15, 128)      89728       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 15, 128)      512         conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 15, 128)      512         conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 15, 128)      512         conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, 15, 128)      0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_2 (Activation)       (None, 15, 128)      0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_3 (Activation)       (None, 15, 128)      0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 128)          0           activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 128)          0           activation_2[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_3 (GlobalM (None, 128)          0           activation_3[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 384)          0           global_max_pooling1d_1[0][0]     \n",
      "                                                                 global_max_pooling1d_2[0][0]     \n",
      "                                                                 global_max_pooling1d_3[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 384)          0           concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 100)          38500       dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 100)          400         dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_4 (Activation)       (None, 100)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 100)          0           activation_4[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 7)            707         dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 7)            28          dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_5 (Activation)       (None, 7)            0           batch_normalization_5[0][0]      \n",
      "==================================================================================================\n",
      "Total params: 233,555\n",
      "Trainable params: 232,573\n",
      "Non-trainable params: 982\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Intialize `KerasClassificationModel` that composes CNN shallow-and-wide network \n",
    "# (name here as`cnn_model`)\n",
    "cls = KerasClassificationModel(save_path=\"./cnn_model_v0\", \n",
    "                               load_path=\"./cnn_model_v0\", \n",
    "                               embedding_size=embedder.dim,\n",
    "                               n_classes=classes_vocab.len,\n",
    "                               model_name=\"cnn_model\",\n",
    "                               text_size=15, # number of tokens\n",
    "                               kernel_sizes_cnn=[3, 5, 7],\n",
    "                               filters_cnn=128,\n",
    "                               dense_size=100,\n",
    "                               optimizer=\"Adam\",\n",
    "                               learning_rate=0.1,\n",
    "                               learning_rate_decay=0.01,\n",
    "                               loss=\"categorical_crossentropy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# `KerasClassificationModel` assumes one-hotted distribution of classes per sample.\n",
    "# `OneHotter` converts indices to one-hot vectors representation.\n",
    "#  To obtain indices we can use our `classes_vocab` intialized and fitted above\n",
    "onehotter = OneHotter(depth=classes_vocab.len, single_vector=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train for 10 epochs\n",
    "for ep in range(10):\n",
    "    for x, y in train_iterator.gen_batches(batch_size=64, \n",
    "                                           data_type=\"train\"):\n",
    "        x_embed = embedder(tokenizer(str_lower(x)))\n",
    "        y_onehot = onehotter(classes_vocab(y))\n",
    "        cls.train_on_batch(x_embed, y_onehot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:15:22.184 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 386: [saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/cnn_model_v0_opt.json]\n"
     ]
    }
   ],
   "source": [
    "# Save model weights and parameters\n",
    "cls.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Infering on validation data we get probability distribution on given data.\n",
    "y_valid_pred = cls(embedder(tokenizer(str_lower(x_valid))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To convert probability distribution to labels, \n",
    "# we first need to convert probabilities to indices,\n",
    "# and then using vocabulary `classes_vocab` convert indices to labels.\n",
    "# \n",
    "# `Proba2Labels` converts probabilities to indices and supports three different modes:\n",
    "# if `max_proba` is true, returns indices of the highest probabilities\n",
    "# if `confident_threshold` is given, returns indices with probabiltiies higher than threshold\n",
    "# if `top_n` is given, returns `top_n` indices with highest probabilities\n",
    "prob2labels = Proba2Labels(max_proba=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text sample: I need seating at Floating restaurant in Tennessee for a group of 9\n",
      "True label: ['BookRestaurant']\n",
      "Predicted probability distribution: {'GetWeather': 4.443174475454725e-05, 'PlayMusic': 0.0002085473679471761, 'SearchScreeningEvent': 6.492184911621734e-05, 'BookRestaurant': 0.9995043277740479, 'RateBook': 0.00021818796813022345, 'SearchCreativeWork': 0.0013526129769161344, 'AddToPlaylist': 8.029041782720014e-05}\n",
      "Predicted label: ['BookRestaurant']\n"
     ]
    }
   ],
   "source": [
    "# Let's look into obtained result\n",
    "print(\"Text sample: {}\".format(x_valid[0]))\n",
    "print(\"True label: {}\".format(y_valid[0]))\n",
    "print(\"Predicted probability distribution: {}\".format(dict(zip(classes_vocab.keys(), \n",
    "                                                               y_valid_pred[0]))))\n",
    "print(\"Predicted label: {}\".format(classes_vocab(prob2labels(y_valid_pred))[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.982373308152345"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate sets accuracy\n",
    "sets_accuracy(y_valid, classes_vocab(prob2labels(y_valid_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  SklearnComponent classifier on GloVe weighted by TF-IDF embeddings in python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:15:22.961 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 218: Cannot load model from /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v1.pkl\n",
      "2019-02-12 12:15:22.962 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 165: Initializing model sklearn.linear_model:LogisticRegression from scratch\n"
     ]
    }
   ],
   "source": [
    "# initialize sklearn classifier, all parameters for classifier could be passed\n",
    "cls = SklearnComponent(\n",
    "    model_class=\"sklearn.linear_model:LogisticRegression\",\n",
    "    infer_method=\"predict\",\n",
    "    save_path='./logreg_v1.pkl',\n",
    "    load_path='./logreg_v1.pkl',\n",
    "    C=1,\n",
    "    mode='train')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:15:44.521 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 108: Fitting model sklearn.linear_model:LogisticRegression\n",
      "2019-02-12 12:15:46.59 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 240: Saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v1.pkl\n"
     ]
    }
   ],
   "source": [
    "# fit sklearn classifier and save it\n",
    "cls.fit(weighted_embedder(str_lower(tokenizer(x_train))), y_train)\n",
    "cls.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_valid_pred = cls(weighted_embedder(str_lower(tokenizer(x_valid))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text sample: I need seating at Floating restaurant in Tennessee for a group of 9\n",
      "True label: ['BookRestaurant']\n",
      "Predicted label: BookRestaurant\n"
     ]
    }
   ],
   "source": [
    "# Let's look into obtained result\n",
    "print(\"Text sample: {}\".format(x_valid[0]))\n",
    "print(\"True label: {}\".format(y_valid[0]))\n",
    "print(\"Predicted label: {}\".format(y_valid_pred[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9184765502045955"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's calculate sets accuracy (because each element is a list of labels)\n",
    "sets_accuracy(np.squeeze(y_valid), y_valid_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's free our memory from embeddings and models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedder.reset()\n",
    "cls.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models from configs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import build_model\n",
    "from deeppavlov import train_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SklearnComponent classifier on Tfidf-features from config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg_config = {\n",
    "  \"dataset_reader\": {\n",
    "    \"class_name\": \"basic_classification_reader\",\n",
    "    \"x\": \"text\",\n",
    "    \"y\": \"intents\",\n",
    "    \"data_path\": \"./snips\"\n",
    "  },\n",
    "  \"dataset_iterator\": {\n",
    "    \"class_name\": \"basic_classification_iterator\",\n",
    "    \"seed\": 42,\n",
    "    \"split_seed\": 23,\n",
    "    \"field_to_split\": \"train\",\n",
    "    \"split_fields\": [\n",
    "      \"train\",\n",
    "      \"valid\"\n",
    "    ],\n",
    "    \"split_proportions\": [\n",
    "      0.9,\n",
    "      0.1\n",
    "    ]\n",
    "  },\n",
    "  \"chainer\": {\n",
    "    \"in\": [\n",
    "      \"x\"\n",
    "    ],\n",
    "    \"in_y\": [\n",
    "      \"y\"\n",
    "    ],\n",
    "    \"pipe\": [\n",
    "      {\n",
    "        \"id\": \"classes_vocab\",\n",
    "        \"class_name\": \"simple_vocab\",\n",
    "        \"fit_on\": [\n",
    "          \"y\"\n",
    "        ],\n",
    "        \"save_path\": \"./snips/classes.dict\",\n",
    "        \"load_path\": \"./snips/classes.dict\",\n",
    "        \"in\": \"y\",\n",
    "        \"out\": \"y_ids\"\n",
    "      },\n",
    "      {\n",
    "        \"in\": [\n",
    "          \"x\"\n",
    "        ],\n",
    "        \"out\": [\n",
    "          \"x_vec\"\n",
    "        ],\n",
    "        \"fit_on\": [\n",
    "          \"x\",\n",
    "          \"y_ids\"\n",
    "        ],\n",
    "        \"id\": \"tfidf_vec\",\n",
    "        \"class_name\": \"sklearn_component\",\n",
    "        \"save_path\": \"tfidf_v1.pkl\",\n",
    "        \"load_path\": \"tfidf_v1.pkl\",\n",
    "        \"model_class\": \"sklearn.feature_extraction.text:TfidfVectorizer\",\n",
    "        \"infer_method\": \"transform\"\n",
    "      },\n",
    "      {\n",
    "        \"in\": \"x\",\n",
    "        \"out\": \"x_tok\",\n",
    "        \"id\": \"my_tokenizer\",\n",
    "        \"class_name\": \"nltk_moses_tokenizer\",\n",
    "        \"tokenizer\": \"wordpunct_tokenize\"\n",
    "      },\n",
    "      {\n",
    "        \"in\": [\n",
    "          \"x_vec\"\n",
    "        ],\n",
    "        \"out\": [\n",
    "          \"y_pred\"\n",
    "        ],\n",
    "        \"fit_on\": [\n",
    "          \"x_vec\",\n",
    "          \"y\"\n",
    "        ],\n",
    "        \"class_name\": \"sklearn_component\",\n",
    "        \"main\": True,\n",
    "        \"save_path\": \"logreg_v2.pkl\",\n",
    "        \"load_path\": \"logreg_v2.pkl\",\n",
    "        \"model_class\": \"sklearn.linear_model:LogisticRegression\",\n",
    "        \"infer_method\": \"predict\",\n",
    "        \"ensure_list_output\": True\n",
    "      }\n",
    "    ],\n",
    "    \"out\": [\n",
    "      \"y_pred\"\n",
    "    ]\n",
    "  },\n",
    "  \"train\": {\n",
    "    \"batch_size\": 64,\n",
    "    \"metrics\": [\n",
    "      \"accuracy\"\n",
    "    ],\n",
    "    \"validate_best\": True,\n",
    "    \"test_best\": False\n",
    "  }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:15:52.310 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 96: Cannot find /home/vimary/ipavlov/Pilot/examples/tutorials/snips/valid.csv file\n",
      "2019-02-12 12:15:52.310 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 96: Cannot find /home/vimary/ipavlov/Pilot/examples/tutorials/snips/test.csv file\n",
      "2019-02-12 12:15:52.311 INFO in 'deeppavlov.dataset_iterators.basic_classification_iterator'['basic_classification_iterator'] at line 73: Splitting field <<train>> to new fields <<['train', 'valid']>>\n",
      "2019-02-12 12:15:52.314 WARNING in 'deeppavlov.core.commands.train'['train'] at line 108: \"validate_best\" and \"test_best\" parameters are deprecated. Please, use \"evaluation_targets\" list instead\n",
      "2019-02-12 12:15:52.322 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:15:52.339 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 89: [saving vocabulary to /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:15:52.340 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 218: Cannot load model from /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v1.pkl\n",
      "2019-02-12 12:15:52.341 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 165: Initializing model sklearn.feature_extraction.text:TfidfVectorizer from scratch\n",
      "2019-02-12 12:15:52.389 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 108: Fitting model sklearn.feature_extraction.text:TfidfVectorizer\n",
      "2019-02-12 12:15:52.493 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 240: Saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v1.pkl\n",
      "2019-02-12 12:15:52.510 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 218: Cannot load model from /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v2.pkl\n",
      "2019-02-12 12:15:52.510 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 165: Initializing model sklearn.linear_model:LogisticRegression from scratch\n",
      "2019-02-12 12:15:53.67 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 108: Fitting model sklearn.linear_model:LogisticRegression\n",
      "2019-02-12 12:15:53.254 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 240: Saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v2.pkl\n",
      "2019-02-12 12:15:53.255 WARNING in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 295: Using NNTrainer for a pipeline without batched training\n",
      "2019-02-12 12:15:53.256 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 240: Saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v2.pkl\n",
      "2019-02-12 12:15:53.257 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:15:53.258 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.feature_extraction.text:TfidfVectorizer from /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v1.pkl\n",
      "2019-02-12 12:15:53.263 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.feature_extraction.textTfidfVectorizer loaded  with parameters\n",
      "2019-02-12 12:15:53.264 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n",
      "2019-02-12 12:15:53.266 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.linear_model:LogisticRegression from /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v2.pkl\n",
      "2019-02-12 12:15:53.266 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.linear_model.logisticLogisticRegression loaded  with parameters\n",
      "2019-02-12 12:15:53.267 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n",
      "2019-02-12 12:15:53.346 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:15:53.347 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.feature_extraction.text:TfidfVectorizer from /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v1.pkl\n",
      "2019-02-12 12:15:53.352 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.feature_extraction.textTfidfVectorizer loaded  with parameters\n",
      "2019-02-12 12:15:53.352 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n",
      "2019-02-12 12:15:53.354 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.linear_model:LogisticRegression from /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v2.pkl\n",
      "2019-02-12 12:15:53.354 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.linear_model.logisticLogisticRegression loaded  with parameters\n",
      "2019-02-12 12:15:53.355 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"accuracy\": 0.983}, \"time_spent\": \"0:00:01\"}}\n"
     ]
    }
   ],
   "source": [
    "# we can train and evaluate model from config\n",
    "m = train_model(logreg_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:15:53.359 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:15:53.360 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.feature_extraction.text:TfidfVectorizer from /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v1.pkl\n",
      "2019-02-12 12:15:53.366 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.feature_extraction.textTfidfVectorizer loaded  with parameters\n",
      "2019-02-12 12:15:53.367 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n",
      "2019-02-12 12:15:53.368 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.linear_model:LogisticRegression from /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v2.pkl\n",
      "2019-02-12 12:15:53.369 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.linear_model.logisticLogisticRegression loaded  with parameters\n",
      "2019-02-12 12:15:53.369 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n"
     ]
    }
   ],
   "source": [
    "# or we can just load pre-trained model (conicides with what we did above)\n",
    "m = build_model(logreg_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['GetWeather']]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m([\"Is it freezing in Offerman, California?\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### KerasClassificationModel on GloVe embeddings from config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "cnn_config = {\n",
    "  \"dataset_reader\": {\n",
    "    \"class_name\": \"basic_classification_reader\",\n",
    "    \"x\": \"text\",\n",
    "    \"y\": \"intents\",\n",
    "    \"data_path\": \"snips\"\n",
    "  },\n",
    "  \"dataset_iterator\": {\n",
    "    \"class_name\": \"basic_classification_iterator\",\n",
    "    \"seed\": 42,\n",
    "    \"split_seed\": 23,\n",
    "    \"field_to_split\": \"train\",\n",
    "    \"split_fields\": [\n",
    "      \"train\",\n",
    "      \"valid\"\n",
    "    ],\n",
    "    \"split_proportions\": [\n",
    "      0.9,\n",
    "      0.1\n",
    "    ]\n",
    "  },\n",
    "  \"chainer\": {\n",
    "    \"in\": [\n",
    "      \"x\"\n",
    "    ],\n",
    "    \"in_y\": [\n",
    "      \"y\"\n",
    "    ],\n",
    "    \"pipe\": [\n",
    "      {\n",
    "        \"id\": \"classes_vocab\",\n",
    "        \"class_name\": \"simple_vocab\",\n",
    "        \"fit_on\": [\n",
    "          \"y\"\n",
    "        ],\n",
    "        \"level\": \"token\",\n",
    "        \"save_path\": \"./snips/classes.dict\",\n",
    "        \"load_path\": \"./snips/classes.dict\",\n",
    "        \"in\": \"y\",\n",
    "        \"out\": \"y_ids\"\n",
    "      },\n",
    "      {\n",
    "        \"in\": \"x\",\n",
    "        \"out\": \"x_tok\",\n",
    "        \"id\": \"my_tokenizer\",\n",
    "        \"class_name\": \"nltk_tokenizer\",\n",
    "        \"tokenizer\": \"wordpunct_tokenize\"\n",
    "      },\n",
    "      {\n",
    "        \"in\": \"x_tok\",\n",
    "        \"out\": \"x_emb\",\n",
    "        \"id\": \"my_embedder\",\n",
    "        \"class_name\": \"glove\",\n",
    "        \"load_path\": \"./glove.6B.100d.txt\",\n",
    "        \"dim\": 100,\n",
    "        \"pad_zero\": True\n",
    "      },\n",
    "      {\n",
    "        \"in\": \"y_ids\",\n",
    "        \"out\": \"y_onehot\",\n",
    "        \"class_name\": \"one_hotter\",\n",
    "        \"depth\": \"#classes_vocab.len\",\n",
    "        \"single_vector\": True\n",
    "      },\n",
    "      {\n",
    "        \"in\": [\n",
    "          \"x_emb\"\n",
    "        ],\n",
    "        \"in_y\": [\n",
    "          \"y_onehot\"\n",
    "        ],\n",
    "        \"out\": [\n",
    "          \"y_pred_probas\"\n",
    "        ],\n",
    "        \"main\": True,\n",
    "        \"class_name\": \"keras_classification_model\",\n",
    "        \"save_path\": \"./cnn_model_v1\",\n",
    "        \"load_path\": \"./cnn_model_v1\",\n",
    "        \"embedding_size\": \"#my_embedder.dim\",\n",
    "        \"n_classes\": \"#classes_vocab.len\",\n",
    "        \"kernel_sizes_cnn\": [\n",
    "          1,\n",
    "          2,\n",
    "          3\n",
    "        ],\n",
    "        \"filters_cnn\": 256,\n",
    "        \"optimizer\": \"Adam\",\n",
    "        \"learning_rate\": 0.01,\n",
    "        \"learning_rate_decay\": 0.1,\n",
    "        \"loss\": \"categorical_crossentropy\",\n",
    "        \"coef_reg_cnn\": 1e-4,\n",
    "        \"coef_reg_den\": 1e-4,\n",
    "        \"dropout_rate\": 0.5,\n",
    "        \"dense_size\": 100,\n",
    "        \"model_name\": \"cnn_model\"\n",
    "      },\n",
    "      {\n",
    "        \"in\": \"y_pred_probas\",\n",
    "        \"out\": \"y_pred_ids\",\n",
    "        \"class_name\": \"proba2labels\",\n",
    "        \"max_proba\": True\n",
    "      },\n",
    "      {\n",
    "        \"in\": \"y_pred_ids\",\n",
    "        \"out\": \"y_pred_labels\",\n",
    "        \"ref\": \"classes_vocab\"\n",
    "      }\n",
    "    ],\n",
    "    \"out\": [\n",
    "      \"y_pred_labels\"\n",
    "    ]\n",
    "  },\n",
    "  \"train\": {\n",
    "    \"epochs\": 10,\n",
    "    \"batch_size\": 64,\n",
    "    \"metrics\": [\n",
    "      \"sets_accuracy\",\n",
    "      \"f1_macro\",\n",
    "      {\n",
    "        \"name\": \"roc_auc\",\n",
    "        \"inputs\": [\"y_onehot\", \"y_pred_probas\"]\n",
    "      }\n",
    "    ],\n",
    "    \"validation_patience\": 5,\n",
    "    \"val_every_n_epochs\": 1,\n",
    "    \"log_every_n_epochs\": 1,\n",
    "    \"show_examples\": True,\n",
    "    \"validate_best\": True,\n",
    "    \"test_best\": False\n",
    "  }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:15:54.311 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 96: Cannot find /home/vimary/ipavlov/Pilot/examples/tutorials/snips/valid.csv file\n",
      "2019-02-12 12:15:54.312 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 96: Cannot find /home/vimary/ipavlov/Pilot/examples/tutorials/snips/test.csv file\n",
      "2019-02-12 12:15:54.313 INFO in 'deeppavlov.dataset_iterators.basic_classification_iterator'['basic_classification_iterator'] at line 73: Splitting field <<train>> to new fields <<['train', 'valid']>>\n",
      "2019-02-12 12:15:54.316 WARNING in 'deeppavlov.core.commands.train'['train'] at line 108: \"validate_best\" and \"test_best\" parameters are deprecated. Please, use \"evaluation_targets\" list instead\n",
      "2019-02-12 12:15:54.319 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:15:54.335 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 89: [saving vocabulary to /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:15:54.337 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 52: [loading GloVe embeddings from `/home/vimary/ipavlov/Pilot/examples/tutorials/glove.6B.100d.txt`]\n",
      "2019-02-12 12:16:14.207 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 272: [initializing `KerasClassificationModel` from scratch as cnn_model]\n",
      "2019-02-12 12:16:14.548 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 136: Model was successfully initialized!\n",
      "Model summary:\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, None, 100)    0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, None, 256)    25856       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_2 (Conv1D)               (None, None, 256)    51456       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, None, 256)    77056       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, None, 256)    1024        conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, None, 256)    1024        conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, None, 256)    1024        conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, None, 256)    0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_2 (Activation)       (None, None, 256)    0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_3 (Activation)       (None, None, 256)    0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 256)          0           activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 256)          0           activation_2[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_3 (GlobalM (None, 256)          0           activation_3[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 768)          0           global_max_pooling1d_1[0][0]     \n",
      "                                                                 global_max_pooling1d_2[0][0]     \n",
      "                                                                 global_max_pooling1d_3[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 768)          0           concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 100)          76900       dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 100)          400         dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_4 (Activation)       (None, 100)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 100)          0           activation_4[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 7)            707         dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 7)            28          dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_5 (Activation)       (None, 7)            0           batch_normalization_5[0][0]      \n",
      "==================================================================================================\n",
      "Total params: 235,475\n",
      "Trainable params: 233,725\n",
      "Non-trainable params: 1,750\n",
      "__________________________________________________________________________________________________\n",
      "/home/vimary/tensorflow/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "2019-02-12 12:16:14.932 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 163: New best sets_accuracy of 0.1479\n",
      "2019-02-12 12:16:14.932 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 165: Saving model\n",
      "2019-02-12 12:16:14.933 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 386: [saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/cnn_model_v1_opt.json]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.1479, \"f1_macro\": 0.044, \"roc_auc\": 0.5499}, \"time_spent\": \"0:00:01\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 0, \"batches_seen\": 0, \"train_examples_seen\": 0, \"impatience\": 0, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:16:19.387 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 163: New best sets_accuracy of 0.9434\n",
      "2019-02-12 12:16:19.388 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 165: Saving model\n",
      "2019-02-12 12:16:19.388 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 386: [saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/cnn_model_v1_opt.json]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"train\": {\"eval_examples_count\": 64, \"metrics\": {\"sets_accuracy\": 0.9375, \"f1_macro\": 0.9421, \"roc_auc\": 0.9938}, \"time_spent\": \"0:00:05\", \"examples\": [{\"x\": \"Please find me the work, Instrumental Directions.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"What weather will it be in Battlement Mesa?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"play theme by Yanni on Vimeo\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"rate the Beyond Black saga a one\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find the schedule for The Tooth Will Out at sunrise.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Rate Lords of the Rim zero stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"play an Masaki Aiba tune\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I need a table for 5 at the restaurant I ate at last Oct.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"book a table at a restaurant in Lucerne Valley that serves chicken nugget\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"add the tune to my viajes playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Play some thrash metal.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Need to find the soundtrack called Fire in the Valley\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Get Jump Down painting\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Is it chillier in Baconton KY\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find I Could Fall in Love.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Evolution and the Theory of Games gets a five out of 6.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Is Outlaw of Gor showing at thenearest movie house at 5 A.M.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Rate Pillar of Fire and Other Plays a three\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Play something by Holly Cole on lastfm \", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I need some ambient music. \", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Rate Steps two out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the movie times at the Loews Cineplex\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Find THUNDER IN THE EAST.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this album  zero stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"play music from the sixties\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Find a television show called Twisted.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Show The Late Great Townes Van Zandt\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Will it get hotter around elevenses in KS?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play a cohesive playlist for me\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will there be rainfall at one PM in Catahoula\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play the greatest record by Leroi Moore\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"this essay should get 1 of the points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"book a table in Connecticut in Robinette for one second from now\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook 1 of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"how's the weather going to be at fourteen o'clock in Falkland Islands\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"She me the Sons of Satan Praise the Lord picture\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Please book a Uncommon Grounds Coffeehouse restaurant \", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Make me a reservation at Illinois Central Railroad Freight Depot in Singapore with vickie rodriguez, lila reyes and ruby\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Is Love Is a Ball playing right now?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"The Far Side of the World chronicle deserves three out of 6 points.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Book a restaurant for one in AL.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"add tune to my instrumental funk playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate my current book 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Which films are playing at the closest movie house?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate this book series zero out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a painting called Beyond the Neighbourhood.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is Longwave going to be playing?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Show me Dangers of the Canadian Mounted\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"around here find movie schedule for films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I want to see Teeny Little Super Guy at Malco Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Will it be windy in NM?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Locate the best pub in Apache Junction\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"party for 2 in Cleveland\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"find Dickinson Theatres showing From Bondage to Freedom\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add potje met vet to my electronic gaming playlis\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I'm looking for the movie called The Beast that Shouted Love at the Heart of the World.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Play White Noise.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Find Just South of Heaven\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Put Al Jarreau on the ConcentraciГіn playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"add banking violence and the inner life today to my retro gaming playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"find a movie house with Colic: The Movie that is nearest\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I am looking for any creative work with the title of Journal of Pharmacy and Pharmacology\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"add the album by Cham to my Cloud Rap playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Find an album called List of Re: Hamatora episodes.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}], \"epochs_done\": 1, \"batches_seen\": 224, \"train_examples_seen\": 14295, \"loss\": 1.386025922106845}}\n",
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9434, \"f1_macro\": 0.9427, \"roc_auc\": 0.9965}, \"time_spent\": \"0:00:05\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 1, \"batches_seen\": 224, \"train_examples_seen\": 14295, \"impatience\": 0, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:16:21.734 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 163: New best sets_accuracy of 0.9515\n",
      "2019-02-12 12:16:21.735 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 165: Saving model\n",
      "2019-02-12 12:16:21.735 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 386: [saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/cnn_model_v1_opt.json]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"train\": {\"eval_examples_count\": 64, \"metrics\": {\"sets_accuracy\": 0.9688, \"f1_macro\": 0.9623, \"roc_auc\": 0.999}, \"time_spent\": \"0:00:08\", \"examples\": [{\"x\": \"She me movie times\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I'd like a table in a smoking room in a taverna on sep. 23, 2023\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"find a movie called No More Sadface\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"A rating of 5 of 6 points goes to Dickson McCunn trilogy\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"For the book The Mirrored Heavens  I give one of a possiable 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Patagonia, South Africa?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Where can I watch the trailer for Home Economics\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Book a restaurant on san jacinto day in Anderson for me and my colleagues.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"What time is The Man Who Dared playing at the movie theatre?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"What will the temperature be at midnight in NE\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"I need a reservation for two at a diner in Venezuela\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me the forecast for the distant area of ME at three pm\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"I want to eat at the Trout Creek restaurant for 9 people for bougatsa that is the best\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current novel four out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What will the weather be at Noon in Durbin OH?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find the album Follow That Camel\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Can you play some Andrew Cash music on Slacker\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I need a table right now for four in ME\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"play Peja tunes\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"add this track by clem burke to my atmospheric black metal playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to play the game Show Me the Wonder\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need a table for four at a restaurant in AL\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"what is the forecast for Orienta for hotter weather\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"when are animated movies playing at Goodrich Quality Theaters\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Will there be a blizzard in Egypt?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find me the soundtrack called Enter the Chicken\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"find the video game called Turnin Me On\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Check movie schedules and find which animated movies are being aired in the neighborhood\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"renee sanders, marlene and jewel want to go to a gastropub in the spa\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"I want to watch The Original Recordings\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"book a table in Fort Loudon at a restaurant for 5\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"in 1 week is there going to be a depression in Washington\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find the schedule for Grand Canyon Trail.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Where Can I watch Chaos and Desire?\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add the album to the Six string peacefulness playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What movies are playing at the closest cinema\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Search for the Halfway Home TV show\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a painting called The Book of Folly.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"What is the weather forecast here?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"play In The Arms Of God on Zvooq by Nimal Mendis\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it get colder in Cape Fair\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play some seventies track from top Rie Tomosaka\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be temperate near Neylandville\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"play the most popular album on Google Music by sasu ripatti\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"rate the book series Sons of Destiny a five\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What's the movie schedule for B&B Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Show the movie schedules at KB Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Put the bill berry track on elrow Guest List\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Tell me the weather forecast in 4 years and a half in GA\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"will the weather be warm far from Niger at 15 o'clock\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"find the closest cinema for films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I am giving this current book album 0 out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add song to my Pop Brasil\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Find a reservation at a brasserie restaurant nearby SC for a party of ten\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"can you find Leadership in my library, please?\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"give four stars out of 6 to current book\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Let's hear something from Elena Risteska\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Include dschiwan gasparjan in beth's rare groove playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Book a table at a brasserie type restaurant that serves jain for a party of 8\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Book PM Park, Clear Lake, Iowa at 5 am for 6 people.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Tell me the weather forecast for Molino, Washington\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"I want to add a tune to my spanish metalblood playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Can you find me the Back When I Knew It All album?\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate the current novel four of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 2, \"batches_seen\": 448, \"train_examples_seen\": 28590, \"loss\": 1.2655515175844942}}\n",
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9515, \"f1_macro\": 0.9509, \"roc_auc\": 0.9973}, \"time_spent\": \"0:00:08\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 2, \"batches_seen\": 448, \"train_examples_seen\": 28590, \"impatience\": 0, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:16:24.94 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 163: New best sets_accuracy of 0.9553\n",
      "2019-02-12 12:16:24.94 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 165: Saving model\n",
      "2019-02-12 12:16:24.95 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 386: [saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/cnn_model_v1_opt.json]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"train\": {\"eval_examples_count\": 64, \"metrics\": {\"sets_accuracy\": 0.9844, \"f1_macro\": 0.9859, \"roc_auc\": 0.9998}, \"time_spent\": \"0:00:10\", \"examples\": [{\"x\": \"find the trailer for Hit the Ice\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Can I get the movies  showtimes for the closest movie house.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I want to give this book zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"give The Creator zero points out of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find the movie schedules for Cineplex Odeon Corporation.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Get soundtrack of Comprehensive Knowledge Archive Network\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"play Pandora tunes from the fourties\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"A Sport and a Pastime is a solid 5 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I give Life During Wartime a one out of 6.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Rate this essay two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What is the cloud coverage in my current place\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find the movie schedule for animated movies in the neighborhood.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add ali lohan songs in Club Hits\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Play some alternative music on Vimeo\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Tell me if it'll be freezing here in 21 seconds\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add inconfundible to the piano in the background playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Are there any animated movies playing in the neighborhood?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Showtimes for animated movies in Malco Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Show me the movie schedule for movies opening today close by\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Rate this album a 3\\n\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Can you add this tune to the night out playlist?\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"rate this textbook 3 out 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Rate the Michel Strogoff saga four of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want to listen to the soundtrack Bed of Roses\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"What weather will HI have will there be hail twenty one minutes from now\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"what is the forecast for in 1 second at Monte Sereno for freezing temps\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"give Jackass Investing a three\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Give this album a three\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"play 1951 tunes\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Can you please find me Season of Glass?\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"play Going Down To The River on Pandora\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Play 2004 on pandora\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Add radhae unakku kobam aagathadi to my Women of Metal playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Play Alone, Again from Mike Viola\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"For the current essay I rate 1 out of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"will there be a cloud next year in Kewanee\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"what is the weather of Sri Lanka\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"I give the Knife of Dreams saga a 0 of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want to give The Plague Lords of Ruel 0 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Can I see Standing on the Edge of the Noise in the nearest cinema\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Play the top music from Epic Mazur.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Play Suite Sudarmoricaine by Afi on itunes\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a reservation for 6 at a restaurant in Deersville\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"play laura love songs from 1959\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"what is the MT forecast for 22\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play the top caleigh peters.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"rate this novel five points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"book a table for me and bettye at Washington, D.C. Jewish Community Center in Montana\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"add Puzzles Like You in my playlist Reggae\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"book for 3 in U.S. Virgin Islands\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play the top fifty record from Alan Jardine\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a reservation for three at a top-rated sicilian restaurant in Portugal\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"book cornelia and bettie a table at a brasserie restaurant in Colombia\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Will it get warmer in Berkley\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"add the artist to my emotron playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add in the heart of the world to the Epic Gaming playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Find me the movie times\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you give me a local and current movie schedule \", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current chronicle 2 stars.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add the name the magnificent tree to playlist this is Rosana\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Play a new ballad by Valy on Iheart\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book spot for 7 at NH Theressa\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Add Jermaine Fagan to spring music\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I need a table for 7 people at a bar that specialises in being a protein bar.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}], \"epochs_done\": 3, \"batches_seen\": 672, \"train_examples_seen\": 42885, \"loss\": 1.2179965520543712}}\n",
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9553, \"f1_macro\": 0.9546, \"roc_auc\": 0.9977}, \"time_spent\": \"0:00:10\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 3, \"batches_seen\": 672, \"train_examples_seen\": 42885, \"impatience\": 0, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:16:26.435 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 163: New best sets_accuracy of 0.9566\n",
      "2019-02-12 12:16:26.435 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 165: Saving model\n",
      "2019-02-12 12:16:26.436 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 386: [saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/cnn_model_v1_opt.json]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"train\": {\"eval_examples_count\": 64, \"metrics\": {\"sets_accuracy\": 0.9531, \"f1_macro\": 0.9521, \"roc_auc\": 0.999}, \"time_spent\": \"0:00:12\", \"examples\": [{\"x\": \"Book a northeastern brazilian restaurant for 10 am\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate The Life and Loves of a She-Devil 5 out of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"what is the forecast for Montana at dinner\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Where is The Toxic Avenger II playing\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Play some music on Last Fm\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Tell me the weather forecast one year from now in Kulpsville, Togo\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"show freezing forcast now within the same area in North Dakota\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play some music from 1993 on Itunes.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Which animated movies are playing at the nearest movie house?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I want a table for 4 in Florida\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"book spot for my mother in law and I at 18 o'clock\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play some House music\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a spot for me and sonja at a popular pizzeria\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"I'm in the mood to listen to meditative music.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Play a new tune by louis silvers.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"rate this series 5 out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want to listen to the soundtrack Bed of Roses\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Will the sun be out close-by Admiralty Island National Monument?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play Wynton Kelly music on Netflix sort by popular\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Rate Who Moved My Cheese? a one\\n\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Please find me Glass Cloud – Single.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Show me The Courts of Chaos\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Look for Hail Satanas We Are The Black Legions\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Play the greatest soundtrack by Nhat Son on Last Fm.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Can you find me the trailer of the Hippocratic Oath?\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"what movies are at the nearest movie house\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Play Barbra Streisand music from 1997.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Please use pandora to play a record from 1993\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Play the theme music from 1963 by Yuki Koyanagi\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy in Tequesta?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Weather for Coaldale Arkansas \", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Where can I find the album The Adventures of Lolo II\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Add Buddy DeSylva to my this is j balvin playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie times for movies premiering in the neighbourhood \", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Curse song to my playlist Guest List Engadget\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"book a table 1 year from now for corinne, tisha and I at a restaurant in Guernsey that is top-rated\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"I'd like to watch Wish You Were Dead\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Add to my list  the tune summer of love\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to listen to Roger Daltrey from the sixties on slacker\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"rate this novel a two\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"book me a reservation at a highly rated tavern in Hornersville\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Give the current chronicle 2 stars.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Show me the Caribbean Blue television show\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Play a song from the thirties by Bruno Pelletier\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Open Vimeo and play music.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Rate the current album a 5 out of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Book me a table for one at Blue Ribbon Barbecue\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"I need to book a restaurant with a smoking room in AL\\n\\n\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Will it be hotter neighboring ME on august eighteenth, 2025?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Will it get colder in Alaska?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Will it snow in AMy\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find the schedule for Vanishing of the Bees at a movie house.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"find To Each His Own Cinema, an album\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"add Funtwo to disco fever track\\n\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Can you play some music from my road trip album\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Can I get the movie schedule for the Bow Tie Cinemas.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"What is the movie schedules for films in the neighborhood\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate this book a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Show me the album Til the Morning\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Add album to my Country Hits\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"give one rank to this album\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add this Roy Orbison song onto Women of Comedy\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Play the newest Roger Troutman track possible\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a restaurant at sixteen o'clock in SC\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}], \"epochs_done\": 4, \"batches_seen\": 896, \"train_examples_seen\": 57180, \"loss\": 1.1904606600957257}}\n",
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9566, \"f1_macro\": 0.9559, \"roc_auc\": 0.9978}, \"time_spent\": \"0:00:12\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 4, \"batches_seen\": 896, \"train_examples_seen\": 57180, \"impatience\": 0, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:16:28.776 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 163: New best sets_accuracy of 0.9585\n",
      "2019-02-12 12:16:28.776 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 165: Saving model\n",
      "2019-02-12 12:16:28.777 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 386: [saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/cnn_model_v1_opt.json]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"train\": {\"eval_examples_count\": 64, \"metrics\": {\"sets_accuracy\": 0.9688, \"f1_macro\": 0.9702, \"roc_auc\": 1.0}, \"time_spent\": \"0:00:15\", \"examples\": [{\"x\": \"Play Pandora on Last Fm\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"add this artist to my SinfonГ­a Hipster\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play some movement by Franky Gee\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"When and where is Nefertiti, Queen of the Nile playing?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"What movies are playing at Loews Cineplex?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Include hohenfriedberger marsch to my Novedades Pop list.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Find movie schedules at IMAX Corporation\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Rate A Tale of Love and Darkness 0 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Play ClГЎsicos del Hip Hop EspaГ±ol\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book for 10 in a restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the Return to Grace saga\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"What is the movie schedule 1 second from now\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add tune to my this is animal collective\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Play new track from the fifties\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"add Blag Dahlia to Pura Vida\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Book a restasurant in Pohick Delaware.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"show creative photograph of Icewind Dale: Heart of Winter\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I want to book a delicatessen serving testaroli in Somalia for 7/25/2027.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Book spot in Fults in Federated States Of Micronesia\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"can you find me a showing for Before the Music Dies in one second ?\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"find Remedial Chaos Theory, a soundtrack\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Show me the photograph A Woman from the Street\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Add this artist to my This Is Philip Glass playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"i need to book a table for three in Lesotho\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Book me a restaurant reservation at 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Need a table for sep. first in Haiti for a party of three\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"book a restaurant for three on feb. 18\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"I'd like to watch movies at Amco Entertainment\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"play the Gary Chaw album\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Add this tomoyasu hotei song to my concentraciГіn playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate the current textbook 1 of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to listen to Space music\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"find Dickinson Theatres showing From Bondage to Freedom\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"will it rain on Jan. 18th, 2029 in Kanopolis Arkansas\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find the schedule for animated movies nearby\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"find Plitt Theatres movie schedules\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Add this tune by Rafet el Roman to my Latin Pop Rising playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Book a reservation for a southern brazilian restaurant for 10 people within walking distance of Broadway-Lafayette St\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play music from the list Indie Electronics\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Give this book a rating of four out of 6.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"my feelin' good playlist needs some Mai Selim in it. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like to book a restaurant in Poncha Springs for 8 at 00:32 am\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate this current essay a 5.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a restaurant for marylou and I within walking distance of my mum's hotel\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"find a movie theatre showing The Tailor of Panama\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show creative game Elements of Life: Remixed\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Play the best music by Arthur Johnston.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"add Diarios de Bicicleta to my la la playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to add michelle heaton to this is chopin\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Is it going to be warm here for brunch\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add an album to my week end playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please play something that's freak folk on Google Music\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Play the most popular Johnny Clarke on Deezer.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Please find me the work, Instrumental Directions.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find me the novel called Ressha Sentai ToQger\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"what animated movies are at the nearest movie house\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Play breed the killers on Itunes\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a restaurant that serves rolled oyster in Merkel\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"What will the humidity be like on june eighteenth in my current location\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Tell me when it'll be cloudy in Woodport\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"show movie times\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what time is Bordertown Trail showing\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a table for one in a bar serving saucisse for meal in Calistoga CO\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Will there be snowfall in KY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}], \"epochs_done\": 5, \"batches_seen\": 1120, \"train_examples_seen\": 71475, \"loss\": 1.1707027325672763}}\n",
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9585, \"f1_macro\": 0.9579, \"roc_auc\": 0.998}, \"time_spent\": \"0:00:15\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 5, \"batches_seen\": 1120, \"train_examples_seen\": 71475, \"impatience\": 0, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:16:31.141 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 163: New best sets_accuracy of 0.9604\n",
      "2019-02-12 12:16:31.141 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 165: Saving model\n",
      "2019-02-12 12:16:31.142 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 386: [saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/cnn_model_v1_opt.json]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"train\": {\"eval_examples_count\": 64, \"metrics\": {\"sets_accuracy\": 0.9844, \"f1_macro\": 0.9808, \"roc_auc\": 0.9935}, \"time_spent\": \"0:00:17\", \"examples\": [{\"x\": \"Will it be freezing on 4/20/2038 in AMerican Beach NC\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"put live and rare into dancehall official\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What is the weather like in Wyatte\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate The Descendants two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find the movie schedule for animated movies in the neighbourhood.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"What time is The Bride’s Journey playing at Star Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Help me find the saga titled The Eternal Return\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"add iemand als jij to my playlist named In The Name Of Blues\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Make me a reservation in Hardesty at a joint the is indoor\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play a song from 2003\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"play Disney Sing It! – High School Musical 3: Senior Year\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I want to book a restaurant in 40 weeks in Iowa.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Will it be hot in Keachi\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate this textbook a one\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What time is A Man for Burning playing\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate this book three points out of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"give Heartland chronicle four points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Turn on Spotify to Tiny Tim ep\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"The Postman is awful and only gets a 1 out of 6. \", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Will the weather this week be warmer in Crystal River?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate this novel four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"In Wynnedale AK will it blizzard\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Please give me the movie schedule for Pacific Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Find me the novel of A Dictionary of Slang and Unconventional English\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Tell me when sunrise is in Tennessee\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Please search for Columbia Records 1958вЂ“1986.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find the movie schedule close by\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you play some music by andrew diamond\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Is The Strange Case of the End of Civilization as We Know It playing at the movie theatre\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Rate The Astonishing Life of Octavian Nothing, Traitor to the Nation, Volume II: The Kingdom on the Waves series 2 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want to see Wenn Lucy springt now at a movie theatre.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"give 0 out of 6 points to current book\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What will the weather be in Rwanda?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"add the artist a j pero to my Country Gold playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"For this series I give the rating of four of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Locate the Koi to Senkyo to Chocolate television show\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"What's the forecast for Pipe Spring National Monument?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"this winter, meredith, betty and erika want to food at a gastropub that is in the same area as fran's location.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Give me the schedule now at the nearest movie house\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I want to book a restaurant in Reily VT.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"How cloudy is it in Morrisonville, Kentucky\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play some noise music on Netflix.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Rate this textbook 4 out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What time are movies showing at Megaplex Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a spot at a highly rated afghan restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"What time is The Bride Wore Boots playing\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I give Ruled Britannia a rating of five out of 6.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want to play the game The Carny\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"add agua y sal in Classic Jazz Funk\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Give the current album 1 star\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a book called The Mad Magician\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"What's the weather in Gabon\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Can you play a top song from a chyi chin concerto\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Play the album 21st Century Live by Chet Lam on Itunes.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I need some Hardcore Hip Hop\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Put gregory douglass in Halloween Teens please\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Find the movie schedule for ArcLight Hollywood.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Book something for my girlfriend and I at a food truck that has pizzas in Brookwood on October fifteenth, 2020\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Can you put Musiri Subramania Iyer's song onto the lo-fi love soundtrack?\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"what is the weather forecast for Cuba at eleven am\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Go to the movie The Best of Pirates of the Mississippi\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"search for a photograph of Road Hogs\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Give The Turning Point a 0 out of 6.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find the movie schedule\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}], \"epochs_done\": 6, \"batches_seen\": 1344, \"train_examples_seen\": 85770, \"loss\": 1.153284895100764}}\n",
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9604, \"f1_macro\": 0.9598, \"roc_auc\": 0.9981}, \"time_spent\": \"0:00:17\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 6, \"batches_seen\": 1344, \"train_examples_seen\": 85770, \"impatience\": 0, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:16:33.547 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 163: New best sets_accuracy of 0.9622\n",
      "2019-02-12 12:16:33.548 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 165: Saving model\n",
      "2019-02-12 12:16:33.548 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 386: [saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/cnn_model_v1_opt.json]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"train\": {\"eval_examples_count\": 64, \"metrics\": {\"sets_accuracy\": 0.9375, \"f1_macro\": 0.9374, \"roc_auc\": 0.997}, \"time_spent\": \"0:00:19\", \"examples\": [{\"x\": \"How much wind will there be in NM on november 11th\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"find The Many Loves of Dobie Gillis\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Put Jazzy B on Lazy Chill Afternoon playlist\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What time is The Bride from Hell playing at Malco Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I am giving the book After Henry a rating of 0 out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I need to add an artist to one of my playlists, Classical New Releases Spotify Picks.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Will it be warm here in one hour\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Is it freezing in Kelso\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"give The Story of the Last Thought a five\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add Nazad, nazad, Kalino mome to Escapada\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add outside the dream syndicate to millicent's fresh electronic playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate this album two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Book a reservation for me and my step sister in Nebraska in two seconds\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"I want to play music from 1979 on Groove Shark.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a restaurant for tortelloni for eight\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Add Shooter Jennings to the All Out 70s playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"find the book Metallica Through the Never\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Add Hanging On to my just dance by aftercluv playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play a good John Maher record with Netflix\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I want to rate my current book three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Which animated movies are showing close by?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give 4 stars to the current essay\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want to see Sympathy for the Devil\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"She me movie times at Mann Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Looking for the saga called The Scofflaw\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Will there be a snowstorm in Pomona, New Mexico?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"What will the weather be in Deer River?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"What's the weather in South Punta Gorda Heights\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find movie times for Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Make me a reservation in Colorado at nine am at National Cash Register Building\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play Elitsa Todorova music\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"add this artist to the playlist cool jazz\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"rate this album one out of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What will the weather be like at my current spot on january the 19th\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Please look up the television show, Noel Hill & Tony Linnane.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Can you put this tune onto Latin Dance Cardio?\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate current novel two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Play some sixties on netflix\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Find a soundtrack called The Dragon.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Play Trace Adkins' music from the thirties.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Can i see the Boat People?\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Add the song to my R&B Movement playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"When is Robotix playing?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Find me An Echo in the Darkness\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Where can I watch the television show called Fangs of the Arctic?\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"rate this book 5 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"give one out of 6 stars to Free Market Fairness\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add nina hagen to essential folk\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play The Edge by Deezer on Vans Warped Tour Compilation 2003\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I need a reservation for January 9 at a restaurant that serves souvlaki nearby Cypress Av for a party of 1\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find time for College Rock Stars at any movie theatre\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Elvis Presley and America in my playlist Electro Workout\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Give one 6 stars to this book\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add Michael Hayvoronsky to Lo Que Suena Los Angeles\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"How do I rate this book 4 stars?\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"want to eat somewhere windy in NM\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Open Netflix and find a movie with the song heartful\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"What film is playing nearby\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Put any 1972 record on.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Rate the Beyond This Place chronicle three of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"will Dick Tracy e il gas misterioso start twenty one hours from now\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"play a sound track by Vegard Sverre Tveitan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Play the song Victim Of Changes from Hawkshaw Hawkins on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I need to book a restaurant for eight nearby Limerick one year from now that serves jerky \", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}], \"epochs_done\": 7, \"batches_seen\": 1568, \"train_examples_seen\": 100065, \"loss\": 1.1379986719361372}}\n",
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9622, \"f1_macro\": 0.9617, \"roc_auc\": 0.9982}, \"time_spent\": \"0:00:19\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 7, \"batches_seen\": 1568, \"train_examples_seen\": 100065, \"impatience\": 0, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:16:35.979 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 169: Did not improve on the sets_accuracy of 0.9622\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"train\": {\"eval_examples_count\": 64, \"metrics\": {\"sets_accuracy\": 0.9844, \"f1_macro\": 0.9849, \"roc_auc\": 1.0}, \"time_spent\": \"0:00:22\", \"examples\": [{\"x\": \"can you get me the trailer of The Multiversity?\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find the films at ArcLight Hollywood.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Will the weather be temperate 22 minutes from now in Alba\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"I'm looking for a picture titled Rock Painting\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"What's the weather forecast for Haigler?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Plpay my Disco Fever playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Add artist to playlist Epic Gaming\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me Rapid City Muscle Car.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find the schedule for Evening Clothes in 1 second.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add pete shelley to Is It New Wave\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play some fifties tunes by Mike Mccready\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"play the new Feist on deezer\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Find a show called Ichibyōgoto ni Love for You.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Play some Rockwell from around 1996\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Add Jarvis Cocker to my Chillin' on a Dirt Road playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please play With Echoes In The Movement Of Stone by Faith Evans.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Is there a blizzard in Tennessee Colony, KS\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"What is the local movie schedule\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I want to listen to Swing music on Iheart\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"how's the forecast for my current spot\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"my step aunt and I want to go cheese fries at the tavern\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play me something by Funtwo\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I need to add a tune by Amanda Stern to the playlist cloud rap.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please get me the Before Crisis: Final Fantasy VII television show.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Book a popular bar in Chowchilla\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the TV series I Build the Tower \", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find the schedule for Metallica Through the Never.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show creativity of Doomsday Comfort\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"is it going to be foggy in Jewell Cemetery State Historic Site 7 weeks from now\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"play Punk Essentials on Zvooq\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Which films are playing at the closest cinema?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I want to find the video game Masada Anniversary Edition Vol. 3: The Unknown Masada\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"rate this album one stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Is The Right to Strike playing at Star Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"play Latin Dinner\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Rate this current essay a 5.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want to hear that tune from 2010\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Play thirties concerto music on Google Music\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"show movie times at sunrise\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please look up for the work titled We Own The Night.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Play a song from Helena Iren Michaelsen on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"rate the current book two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Play Elizeth Cardoso to my Nothing But A Party R&B playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Play the most popular music by Ronald Isley on Google Music\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I want to hear something eclectic\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Go to the saga The Quantum Thief\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I'd like to see weather conditions for Ennis.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play some songs from the fifties\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"play By The Sleepy Lagoon by Greg Kurstin\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I want to book a restaurant in the same area where I live in MA for ebony and yolanda.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"What is the weather forecast for Agate Fossil Beds National Monument\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"add this track to my global funk\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play Zvooq\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"I'd like to rate this textbook 4 out of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Book a reservation for one at a highly rated restaurant in Datil\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"play a symphony that is good from 2000\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"What is the movie schedule today at Neighborhood Cinema Group?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I'd like to eat salads at a restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"find Plitt Theatres movie schedules\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"I think this novel only deserves 2 points out of 6.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Is it rainy at the Edward L. Ryerson Conservation Area?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Will it be chillier on october 17 nearby East Glacier Park\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Give the current series a rating of three.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"book an oyster bar in AMerican Samoa for lunch\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}], \"epochs_done\": 8, \"batches_seen\": 1792, \"train_examples_seen\": 114360, \"loss\": 1.127019821533135}}\n",
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9622, \"f1_macro\": 0.9617, \"roc_auc\": 0.9983}, \"time_spent\": \"0:00:22\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 8, \"batches_seen\": 1792, \"train_examples_seen\": 114360, \"impatience\": 1, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:16:38.311 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 163: New best sets_accuracy of 0.9629\n",
      "2019-02-12 12:16:38.312 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 165: Saving model\n",
      "2019-02-12 12:16:38.312 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 386: [saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/cnn_model_v1_opt.json]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"train\": {\"eval_examples_count\": 64, \"metrics\": {\"sets_accuracy\": 0.9844, \"f1_macro\": 0.9837, \"roc_auc\": 0.9983}, \"time_spent\": \"0:00:24\", \"examples\": [{\"x\": \"Rate my current essay 1 out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What's the weather in FL?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"play me some Dom Pachino\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Is cloudy in Lyncourt?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Is temperature in Hanksville freezing ?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"play some Bertine Zetlitz record\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"play latest George Ducas music\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"For The Curious Incident of the Dog in the Nightdress I rate it 2 of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"book verdure serving restaurant in Bloom City\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"I want another song in my rock espaГ±ol playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Give the current essay five points / 6.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Play show of Cissy Houston\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Add the artist Gwenno Pipette to the sexy as folk playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I rate Egg Collecting and Bird Life of Australia a zero out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Book me a tibetan restaurant for my boss and I.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"I give The Logic of Sense a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate The Ape-Man Within 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"play In The Disco by Danny Hutton\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"What is the weather like right now for Fort Adams?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add Grey Cloudy Lies to the hip hop playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I'd like to hear the song In a Reverie\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Add por tu maldito amor to my orgullo gay\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Where can I buy The Lying Game\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"rate the current novel 0 of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Please find the movie, A Jingle with Jillian.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I want to find a restaurant that has a table for two at 5 AM\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"tell me how Bellwood weather is\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find a table for madge and tami at a faraway joint on Sterling St that serves chicken divan\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"What cinema has the closest movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give The Blue Equinox series 5 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add this album to my hot house playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I need a table booking for a highly rated sardinian pub.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Can you put some monifah on my disco fever playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I'd like to see movie schedules for animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is F.I.S.T. at Malco Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"What is the weather going to be like in Klondike Gold Rush National Historical Park on february the 28th, 2034?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"show creative picture of The Secret Doctrine\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Book a restaurant in CA for my parents and I on oct. the seventeenth\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Need to find the TV series called Administrative Behavior\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find the work I Looked Up\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"rate this album book zero out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want to listen to Merrily We Roll Along by Marko Desantis.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Play the album entitled Se Potrei Avere Te.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Please help me find the video game John Michael Montgomery discography.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Can you find the album SimpleScreenRecorder\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Will it be warm in Kipp Rhode Island one hour and 9 seconds from now?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find the movie times at Bow Tie Cinemas.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Add Gary Valenciano to the power gaming playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to the Leche con Chocolate playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"show me the movie times in the neighbourhood\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give The Street five points.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want to book a restaurant in Ayer for 2 people.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the Endangered Species song\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"book a gibassier serving tavern in Vermont for nine\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Book me a restaurant reservation for a party of 8 ten hours from now\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Is it warm in Albania at noon\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Is it going to get any hotter in Kerrick?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"book a table for ten in Pollock PA\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"book Guenther House for 6 on Oct. 24, 2035 in Waddy\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"play my melodious playlist\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Add Hallucinations of Despair to my this is trey songz playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I need to book a restaurant in Burkettsville in 2 years  for rhoda adams, roxanne and I\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"give me the local movie times\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"open Itunes and play Kenny Cox most popular concerto\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}], \"epochs_done\": 9, \"batches_seen\": 2016, \"train_examples_seen\": 128655, \"loss\": 1.1169953814574651}}\n",
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9629, \"f1_macro\": 0.9623, \"roc_auc\": 0.9983}, \"time_spent\": \"0:00:24\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 9, \"batches_seen\": 2016, \"train_examples_seen\": 128655, \"impatience\": 0, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:16:40.661 INFO in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 169: Did not improve on the sets_accuracy of 0.9629\n",
      "2019-02-12 12:16:40.693 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:16:40.693 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 52: [loading GloVe embeddings from `/home/vimary/ipavlov/Pilot/examples/tutorials/glove.6B.100d.txt`]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"train\": {\"eval_examples_count\": 64, \"metrics\": {\"sets_accuracy\": 1.0, \"f1_macro\": 1.0, \"roc_auc\": 0.9996}, \"time_spent\": \"0:00:26\", \"examples\": [{\"x\": \"book in town for 3 at a restaurant outdoor that is not far\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Need a table for the day after tomorrow in Clarenceville at the Black Rapids Roadhouse\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"What will the weather be like this tuesday in the area neighboring Rendezvous Mountain Educational State Forest?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate The CIA and the Cult of Intelligence a 5.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Is the forecast windy in Nigeria on Nov. the 6th\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Book the nearby Meriton Grand Hotel Tallinn in Missouri.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"please give me the movie schedule\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Book a reservation for 4 for Cherry Hut at Noon\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"I think Memorial Day should have a rating value of 3 and a best rating of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Play some G. V. Prakash Kumar\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"is The Clowns at the nearest cinema \", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"when is Letters from a Porcupine showing at Alamo Drafthouse Cinema\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Play some fun-punk\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book Guenther House for 6 on Oct. 24, 2035 in Waddy\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"what is the forecast in North Carolina\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"what's the movie schedules for in the neighborhood at the movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Book a highly rated food court for 2 people on jul. 4th.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Can you put freddie freeloader on the playlist instrumental madness\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please get me the Just the Hits 2 TV show.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Book a reservation for seven people at Fraser Mansion in IL\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Use Spotify to play Who Was In My Room Last Night?\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"What is the forecast in Lono\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"I'd like to watch Sherlock Holmes à New York at KB Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Let's listen to the most popular Marty Friedman songs on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"weather in Tioga Colorado\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"I give The Monkey and the Tiger a rating of 2 points.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find WxHexEditor.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Book a table for my granddaughter and I at the highly rated restaurant that is close by in Tuvalu.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Add The Maid of Amsterdam to my 80s smash hits\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"find the picture Louder Than Bombs\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Will it rain in Barberville\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find a show called Ichibyōgoto ni Love for You.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Will it be warmer near here on jan. the fifteenth?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate The Travels of Lao Can five out of 6\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What time is Sontha Ooru playing\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Rate The Hindus: An Alternative History 3 of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"play latest George Ducas music\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"find movie schedules for Dickinson Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Rate this book a five \", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find me a restaurant in Pembine Montana\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"What is the movie schedule right now for movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is it going to be foggy at two am in Barberville\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"What will be the forecast for Belarus in the future around sep. the 22nd, 2020?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"I need to find the work Brotherly Love\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find me the Spartan: Total Warrior painting\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I'd like to see the television show Best-Of: Design of a Decade 2003–2013\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Add this album to my spotify orchestra cello playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Give this textbook a rating of three.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Give The Irish Filmography saga a rating of 2 out of 6.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What is the weather forecast for Theodore Roosevelt Inaugural National Historic Site\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"play Isaac Hayes on Pandora from love, sweat and beer ep\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be colder in Oswego 16 weeks from now ?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"What is the movies playing at North American Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"What will the weather be a nine in Willow River State Park?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Find me the photograph The Late Music\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Get me a Johnny Cool photograph\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"add artist my laundry playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Is the forecast colder in Idaho 1 second from now\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"book a spot for 3 at the pizza place\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find me a showing of The Vanquished that starts nine hours and 1 second from now.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please help me find the video game John Michael Montgomery discography.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Search for To Heart 2\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Add Nightmares That Surface from Shallow Sleep to michael's Rock Solid playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Find a show called The Inheritors\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}], \"epochs_done\": 10, \"batches_seen\": 2240, \"train_examples_seen\": 142950, \"loss\": 1.10787156862872}}\n",
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9629, \"f1_macro\": 0.9623, \"roc_auc\": 0.9983}, \"time_spent\": \"0:00:27\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}], \"epochs_done\": 10, \"batches_seen\": 2240, \"train_examples_seen\": 142950, \"impatience\": 1, \"patience_limit\": 5}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:17:00.634 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 302: [initializing `KerasClassificationModel` from saved]\n",
      "2019-02-12 12:17:00.963 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 312: [loading weights from cnn_model_v1.h5]\n",
      "2019-02-12 12:17:01.131 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 136: Model was successfully initialized!\n",
      "Model summary:\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, None, 100)    0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, None, 256)    25856       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_2 (Conv1D)               (None, None, 256)    51456       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, None, 256)    77056       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, None, 256)    1024        conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, None, 256)    1024        conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, None, 256)    1024        conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, None, 256)    0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_2 (Activation)       (None, None, 256)    0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_3 (Activation)       (None, None, 256)    0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 256)          0           activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 256)          0           activation_2[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_3 (GlobalM (None, 256)          0           activation_3[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 768)          0           global_max_pooling1d_1[0][0]     \n",
      "                                                                 global_max_pooling1d_2[0][0]     \n",
      "                                                                 global_max_pooling1d_3[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 768)          0           concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 100)          76900       dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 100)          400         dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_4 (Activation)       (None, 100)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 100)          0           activation_4[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 7)            707         dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 7)            28          dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_5 (Activation)       (None, 7)            0           batch_normalization_5[0][0]      \n",
      "==================================================================================================\n",
      "Total params: 235,475\n",
      "Trainable params: 233,725\n",
      "Non-trainable params: 1,750\n",
      "__________________________________________________________________________________________________\n",
      "2019-02-12 12:17:01.431 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:17:01.431 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 52: [loading GloVe embeddings from `/home/vimary/ipavlov/Pilot/examples/tutorials/glove.6B.100d.txt`]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9629, \"f1_macro\": 0.9623, \"roc_auc\": 0.9983}, \"time_spent\": \"0:00:01\", \"examples\": [{\"x\": \"Book a table at Carter House Inn in Saint Bonaventure, Alaska.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Rate the current textbook one of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find a nearby movie schedule for movies\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"what is the Mississippi for the week\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Play me a song from 1968 on Spotify\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Book a table for me, naomi and elisabeth at a brasserie with wifi\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"The current album gets three out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"find Goodrich Quality Theaters films\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"show me the picture Unfinished Monkey Business\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"When is The Third Eye showing at Dickinson Theatres?\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Please get me the Welcome to the Rileys game.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Find a song called Bronco Billy.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"Rate this essay five stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"add tune to my relax & unwind playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"play 2007 tunes by Bunny Berigan\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"book a table for ten downtown at a close-by restaurant\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Find the schedule for for Corn at eleven A.M. at Loews Cineplex Entertainment.\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"1 minute from now, I will need reservations at a restaurant in Vanlue.\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Play hanging in the balance by Nik Kershaw on Zvooq.\", \"y_predicted\": [\"PlayMusic\"], \"y_true\": [\"PlayMusic\"]}, {\"x\": \"Will it be windy at 4 Pm in NY?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate my current textbook 1 out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"What are the weather conditions in Noel?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Add this artist to the laugh list\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I am rating Book of Challenges four stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"rate this textbook a 4\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Put an album by max richter into my this is Rosana playlist. \", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"where can i watch animated movies around here\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Is A Man, a Woman, and a Bank showing in the nearest Neighborhood Cinema Group\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"book a popular food truck in Kentucky\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"Show me animated movies that are playig at Great Escape Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"add Sara Carter to my Nothing But A Party R&B\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I would like an outdoor cafeteria for 3\", \"y_predicted\": [\"BookRestaurant\"], \"y_true\": [\"BookRestaurant\"]}, {\"x\": \"rate the book Whit a zero\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Find a show called Time Is Just the Same.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchCreativeWork\"]}, {\"x\": \"I need the weather in Hubbardston, will it be chillier?\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"rate the previous essay four of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"Add wiktor coj to the Sleep playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Rate Dixie Lullaby: A Story of Music, Race and New Beginnings in a New South five out of 6 points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"please put live with me onto my playlist named CARГЃCTER LATINO\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Please add tobymac's song onto the indiespensables playlist.\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Show me the movie schedule for Caribbean Cinemas\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Can you put this song on the metal xplorer playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"Add this tune to my rage radio playlist\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"I want to go see A Troll in Central Park.\", \"y_predicted\": [\"SearchCreativeWork\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give the current series a one.\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like to watch animated movies at National Amusements\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"my jazz for loving couples needs more push the button\", \"y_predicted\": [\"AddToPlaylist\"], \"y_true\": [\"AddToPlaylist\"]}, {\"x\": \"What are the movie schedules for Kerasotes Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"rate the Dry series two out of 6 stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I want a list of showings of Days of Fire at Harkins Theatres\", \"y_predicted\": [\"SearchScreeningEvent\"], \"y_true\": [\"SearchScreeningEvent\"]}, {\"x\": \"Give White House Diary two points\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}, {\"x\": \"I'd like the weather forecast in Gang Mills four years from now.\", \"y_predicted\": [\"GetWeather\"], \"y_true\": [\"GetWeather\"]}, {\"x\": \"Rate Tropic of Capricorn two stars\", \"y_predicted\": [\"RateBook\"], \"y_true\": [\"RateBook\"]}]}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:17:21.399 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 302: [initializing `KerasClassificationModel` from saved]\n",
      "2019-02-12 12:17:21.744 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 312: [loading weights from cnn_model_v1.h5]\n",
      "2019-02-12 12:17:21.909 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 136: Model was successfully initialized!\n",
      "Model summary:\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, None, 100)    0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, None, 256)    25856       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_2 (Conv1D)               (None, None, 256)    51456       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, None, 256)    77056       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, None, 256)    1024        conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, None, 256)    1024        conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, None, 256)    1024        conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, None, 256)    0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_2 (Activation)       (None, None, 256)    0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_3 (Activation)       (None, None, 256)    0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 256)          0           activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 256)          0           activation_2[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_3 (GlobalM (None, 256)          0           activation_3[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 768)          0           global_max_pooling1d_1[0][0]     \n",
      "                                                                 global_max_pooling1d_2[0][0]     \n",
      "                                                                 global_max_pooling1d_3[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 768)          0           concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 100)          76900       dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 100)          400         dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_4 (Activation)       (None, 100)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 100)          0           activation_4[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 7)            707         dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 7)            28          dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_5 (Activation)       (None, 7)            0           batch_normalization_5[0][0]      \n",
      "==================================================================================================\n",
      "Total params: 235,475\n",
      "Trainable params: 233,725\n",
      "Non-trainable params: 1,750\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# we can train and evaluate model from config\n",
    "m = train_model(cnn_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:17:21.914 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:17:21.915 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 52: [loading GloVe embeddings from `/home/vimary/ipavlov/Pilot/examples/tutorials/glove.6B.100d.txt`]\n",
      "2019-02-12 12:17:42.89 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 302: [initializing `KerasClassificationModel` from saved]\n",
      "2019-02-12 12:17:42.406 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 312: [loading weights from cnn_model_v1.h5]\n",
      "2019-02-12 12:17:42.569 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 136: Model was successfully initialized!\n",
      "Model summary:\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, None, 100)    0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, None, 256)    25856       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_2 (Conv1D)               (None, None, 256)    51456       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, None, 256)    77056       input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, None, 256)    1024        conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, None, 256)    1024        conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, None, 256)    1024        conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, None, 256)    0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_2 (Activation)       (None, None, 256)    0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_3 (Activation)       (None, None, 256)    0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 256)          0           activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 256)          0           activation_2[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_3 (GlobalM (None, 256)          0           activation_3[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 768)          0           global_max_pooling1d_1[0][0]     \n",
      "                                                                 global_max_pooling1d_2[0][0]     \n",
      "                                                                 global_max_pooling1d_3[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 768)          0           concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 100)          76900       dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 100)          400         dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_4 (Activation)       (None, 100)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 100)          0           activation_4[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 7)            707         dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 7)            28          dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_5 (Activation)       (None, 7)            0           batch_normalization_5[0][0]      \n",
      "==================================================================================================\n",
      "Total params: 235,475\n",
      "Trainable params: 233,725\n",
      "Non-trainable params: 1,750\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# or we can just load pre-trained model (conicides with what we did above)\n",
    "m = build_model(cnn_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['GetWeather']]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m([\"Is it freezing in Offerman, California?\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SklearnComponent classifier on GloVe weighted by TF-IDF embeddings from config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg_config = {\n",
    "  \"dataset_reader\": {\n",
    "    \"class_name\": \"basic_classification_reader\",\n",
    "    \"x\": \"text\",\n",
    "    \"y\": \"intents\",\n",
    "    \"data_path\": \"snips\"\n",
    "  },\n",
    "  \"dataset_iterator\": {\n",
    "    \"class_name\": \"basic_classification_iterator\",\n",
    "    \"seed\": 42,\n",
    "      \"split_seed\": 23,\n",
    "    \"field_to_split\": \"train\",\n",
    "    \"split_fields\": [\n",
    "      \"train\",\n",
    "      \"valid\"\n",
    "    ],\n",
    "    \"split_proportions\": [\n",
    "      0.9,\n",
    "      0.1\n",
    "    ]\n",
    "  },\n",
    "  \"chainer\": {\n",
    "    \"in\": [\n",
    "      \"x\"\n",
    "    ],\n",
    "    \"in_y\": [\n",
    "      \"y\"\n",
    "    ],\n",
    "    \"pipe\": [\n",
    "      {\n",
    "        \"id\": \"classes_vocab\",\n",
    "        \"class_name\": \"simple_vocab\",\n",
    "        \"fit_on\": [\n",
    "          \"y\"\n",
    "        ],\n",
    "        \"save_path\": \"./snips/classes.dict\",\n",
    "        \"load_path\": \"./snips/classes.dict\",\n",
    "        \"in\": \"y\",\n",
    "        \"out\": \"y_ids\"\n",
    "      },\n",
    "      {\n",
    "        \"in\": [\n",
    "          \"x\"\n",
    "        ],\n",
    "        \"out\": [\n",
    "          \"x_vec\"\n",
    "        ],\n",
    "        \"fit_on\": [\n",
    "          \"x\",\n",
    "          \"y_ids\"\n",
    "        ],\n",
    "        \"id\": \"my_tfidf_vectorizer\",\n",
    "        \"class_name\": \"sklearn_component\",\n",
    "        \"save_path\": \"tfidf_v2.pkl\",\n",
    "        \"load_path\": \"tfidf_v2.pkl\",\n",
    "        \"model_class\": \"sklearn.feature_extraction.text:TfidfVectorizer\",\n",
    "        \"infer_method\": \"transform\"\n",
    "      },\n",
    "      {\n",
    "        \"in\": \"x\",\n",
    "        \"out\": \"x_tok\",\n",
    "        \"id\": \"my_tokenizer\",\n",
    "        \"class_name\": \"nltk_moses_tokenizer\"\n",
    "      },\n",
    "      {\n",
    "        \"in\": \"x_tok\",\n",
    "        \"out\": \"x_emb\",\n",
    "        \"id\": \"my_embedder\",\n",
    "        \"class_name\": \"glove\",\n",
    "        \"save_path\": \"./glove.6B.100d.txt\",\n",
    "        \"load_path\": \"./glove.6B.100d.txt\",\n",
    "        \"dim\": 100,\n",
    "        \"pad_zero\": True\n",
    "      },\n",
    "      {\n",
    "        \"class_name\": \"one_hotter\",\n",
    "        \"id\": \"my_onehotter\",\n",
    "        \"depth\": \"#classes_vocab.len\",\n",
    "        \"in\": \"y_ids\",\n",
    "        \"out\": \"y_onehot\",\n",
    "        \"single_vector\": True\n",
    "      },\n",
    "      {\n",
    "        \"in\": \"x_tok\",\n",
    "        \"out\": \"x_weighted_emb\",\n",
    "        \"class_name\": \"tfidf_weighted\",\n",
    "        \"id\": \"my_weighted_embedder\",\n",
    "        \"embedder\": \"#my_embedder\",\n",
    "        \"tokenizer\": \"#my_tokenizer\",\n",
    "        \"vectorizer\": \"#my_tfidf_vectorizer\",\n",
    "          \"mean\": True\n",
    "      },\n",
    "      {\n",
    "        \"in\": [\n",
    "          \"x_weighted_emb\"\n",
    "        ],\n",
    "        \"out\": [\n",
    "          \"y_pred\"\n",
    "        ],\n",
    "        \"fit_on\": [\n",
    "          \"x_weighted_emb\",\n",
    "          \"y\"\n",
    "        ],\n",
    "        \"class_name\": \"sklearn_component\",\n",
    "        \"main\": True,\n",
    "        \"save_path\": \"logreg_v3.pkl\",\n",
    "        \"load_path\": \"logreg_v3.pkl\",\n",
    "        \"model_class\": \"sklearn.linear_model:LogisticRegression\",\n",
    "        \"infer_method\": \"predict\",\n",
    "        \"ensure_list_output\": True\n",
    "      }\n",
    "    ],\n",
    "    \"out\": [\n",
    "      \"y_pred\"\n",
    "    ]\n",
    "  },\n",
    "  \"train\": {\n",
    "    \"epochs\": 10,\n",
    "    \"batch_size\": 64,\n",
    "    \"metrics\": [\n",
    "      \"sets_accuracy\"\n",
    "    ],\n",
    "    \"show_examples\": False,\n",
    "    \"validate_best\": True,\n",
    "    \"test_best\": False\n",
    "  }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:32:01.417 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 96: Cannot find /home/vimary/ipavlov/Pilot/examples/tutorials/snips/valid.csv file\n",
      "2019-02-12 12:32:01.417 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 96: Cannot find /home/vimary/ipavlov/Pilot/examples/tutorials/snips/test.csv file\n",
      "2019-02-12 12:32:01.418 INFO in 'deeppavlov.dataset_iterators.basic_classification_iterator'['basic_classification_iterator'] at line 73: Splitting field <<train>> to new fields <<['train', 'valid']>>\n",
      "2019-02-12 12:32:01.420 WARNING in 'deeppavlov.core.commands.train'['train'] at line 108: \"validate_best\" and \"test_best\" parameters are deprecated. Please, use \"evaluation_targets\" list instead\n",
      "2019-02-12 12:32:01.421 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:32:01.439 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 89: [saving vocabulary to /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:32:01.440 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 218: Cannot load model from /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v2.pkl\n",
      "2019-02-12 12:32:01.441 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 165: Initializing model sklearn.feature_extraction.text:TfidfVectorizer from scratch\n",
      "2019-02-12 12:32:01.486 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 108: Fitting model sklearn.feature_extraction.text:TfidfVectorizer\n",
      "2019-02-12 12:32:01.587 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 240: Saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v2.pkl\n",
      "2019-02-12 12:32:01.603 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 52: [loading GloVe embeddings from `/home/vimary/ipavlov/Pilot/examples/tutorials/glove.6B.100d.txt`]\n",
      "2019-02-12 12:32:21.226 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 218: Cannot load model from /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v3.pkl\n",
      "2019-02-12 12:32:21.227 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 165: Initializing model sklearn.linear_model:LogisticRegression from scratch\n",
      "2019-02-12 12:32:43.431 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 108: Fitting model sklearn.linear_model:LogisticRegression\n",
      "2019-02-12 12:32:45.621 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 240: Saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v3.pkl\n",
      "2019-02-12 12:32:45.626 WARNING in 'deeppavlov.core.trainers.nn_trainer'['nn_trainer'] at line 295: Using NNTrainer for a pipeline without batched training\n",
      "2019-02-12 12:32:45.626 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 240: Saving model to /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v3.pkl\n",
      "2019-02-12 12:32:45.658 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:32:45.659 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.feature_extraction.text:TfidfVectorizer from /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v2.pkl\n",
      "2019-02-12 12:32:45.664 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.feature_extraction.textTfidfVectorizer loaded  with parameters\n",
      "2019-02-12 12:32:45.665 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n",
      "2019-02-12 12:32:45.666 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 52: [loading GloVe embeddings from `/home/vimary/ipavlov/Pilot/examples/tutorials/glove.6B.100d.txt`]\n",
      "2019-02-12 12:33:05.258 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.linear_model:LogisticRegression from /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v3.pkl\n",
      "2019-02-12 12:33:05.259 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.linear_model.logisticLogisticRegression loaded  with parameters\n",
      "2019-02-12 12:33:05.259 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n",
      "2019-02-12 12:33:07.749 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:33:07.750 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.feature_extraction.text:TfidfVectorizer from /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v2.pkl\n",
      "2019-02-12 12:33:07.755 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.feature_extraction.textTfidfVectorizer loaded  with parameters\n",
      "2019-02-12 12:33:07.755 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n",
      "2019-02-12 12:33:07.756 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 52: [loading GloVe embeddings from `/home/vimary/ipavlov/Pilot/examples/tutorials/glove.6B.100d.txt`]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9283}, \"time_spent\": \"0:00:03\"}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:33:27.702 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.linear_model:LogisticRegression from /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v3.pkl\n",
      "2019-02-12 12:33:27.702 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.linear_model.logisticLogisticRegression loaded  with parameters\n",
      "2019-02-12 12:33:27.703 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n"
     ]
    }
   ],
   "source": [
    "# we can train and evaluate model from config\n",
    "m = train_model(logreg_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-02-12 12:33:27.742 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 103: [loading vocabulary from /home/vimary/ipavlov/Pilot/examples/tutorials/snips/classes.dict]\n",
      "2019-02-12 12:33:27.743 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.feature_extraction.text:TfidfVectorizer from /home/vimary/ipavlov/Pilot/examples/tutorials/tfidf_v2.pkl\n",
      "2019-02-12 12:33:27.748 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.feature_extraction.textTfidfVectorizer loaded  with parameters\n",
      "2019-02-12 12:33:27.749 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n",
      "2019-02-12 12:33:27.750 INFO in 'deeppavlov.models.embedders.glove_embedder'['glove_embedder'] at line 52: [loading GloVe embeddings from `/home/vimary/ipavlov/Pilot/examples/tutorials/glove.6B.100d.txt`]\n",
      "2019-02-12 12:33:47.483 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 202: Loading model sklearn.linear_model:LogisticRegression from /home/vimary/ipavlov/Pilot/examples/tutorials/logreg_v3.pkl\n",
      "2019-02-12 12:33:47.484 INFO in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 209: Model sklearn.linear_model.logisticLogisticRegression loaded  with parameters\n",
      "2019-02-12 12:33:47.484 WARNING in 'deeppavlov.models.sklearn.sklearn_component'['sklearn_component'] at line 215: Fitting of loaded model can not be continued. Model can be fitted from scratch.If one needs to continue fitting, please, look at `warm_start` parameter\n"
     ]
    }
   ],
   "source": [
    "# or we can just load pre-trained model (conicides with what we did above)\n",
    "m = build_model(logreg_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['GetWeather']]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m([\"Is it freezing in Offerman, California?\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's free memory\n",
    "del m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus: pre-trained CNN model in DeepPavlov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download model files (`wiki.en.bin` 8Gb embeddings):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "! python -m deeppavlov download intents_snips_big"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate metrics on validation set (no test set provided):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "! python -m deeppavlov evaluate intents_snips_big"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or one can use model from python code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import deeppavlov\n",
    "from deeppavlov import build_model, evaluate_model\n",
    "from deeppavlov.download import deep_download\n",
    "\n",
    "config_path = Path(deeppavlov.__file__).parent.joinpath('configs/classifiers/intents_snips_big.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-12-13 18:44:55.284 DEBUG in 'urllib3.connectionpool'['connectionpool'] at line 205: Starting new HTTP connection (1): files.deeppavlov.ai:80\n",
      "2018-12-13 18:44:55.341 DEBUG in 'urllib3.connectionpool'['connectionpool'] at line 393: http://files.deeppavlov.ai:80 \"GET /datasets/snips_intents/train.csv.md5 HTTP/1.1\" 200 44\n",
      "2018-12-13 18:44:55.346 INFO in 'deeppavlov.download'['download'] at line 115: Skipped http://files.deeppavlov.ai/datasets/snips_intents/train.csv download because of matching hashes\n",
      "2018-12-13 18:44:55.348 DEBUG in 'urllib3.connectionpool'['connectionpool'] at line 205: Starting new HTTP connection (1): files.deeppavlov.ai:80\n",
      "2018-12-13 18:44:55.540 DEBUG in 'urllib3.connectionpool'['connectionpool'] at line 393: http://files.deeppavlov.ai:80 \"GET /deeppavlov_data/classifiers/intents_snips_v10.tar.gz.md5 HTTP/1.1\" 200 193\n",
      "2018-12-13 18:44:55.589 INFO in 'deeppavlov.download'['download'] at line 115: Skipped http://files.deeppavlov.ai/deeppavlov_data/classifiers/intents_snips_v10.tar.gz download because of matching hashes\n",
      "2018-12-13 18:44:55.593 DEBUG in 'urllib3.connectionpool'['connectionpool'] at line 205: Starting new HTTP connection (1): files.deeppavlov.ai:80\n",
      "2018-12-13 18:44:55.629 DEBUG in 'urllib3.connectionpool'['connectionpool'] at line 393: http://files.deeppavlov.ai:80 \"GET /deeppavlov_data/embeddings/wiki.en.bin.md5 HTTP/1.1\" 200 46\n",
      "2018-12-13 18:45:11.617 INFO in 'deeppavlov.download'['download'] at line 115: Skipped http://files.deeppavlov.ai/deeppavlov_data/embeddings/wiki.en.bin download because of matching hashes\n"
     ]
    }
   ],
   "source": [
    "# let's download all the required data - model files, embeddings, vocabularies\n",
    "deep_download(config_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-12-13 18:45:11.621 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 100: [loading vocabulary from /home/dilyara/.deeppavlov/models/classifiers/intents_snips_v10/classes.dict]\n",
      "2018-12-13 18:45:11.632 INFO in 'deeppavlov.models.embedders.fasttext_embedder'['fasttext_embedder'] at line 52: [loading fastText embeddings from `/home/dilyara/.deeppavlov/downloads/embeddings/wiki.en.bin`]\n",
      "2018-12-13 18:45:32.229 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 287: [initializing `KerasClassificationModel` from saved]\n",
      "2018-12-13 18:45:32.554 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 297: [loading weights from model.h5]\n",
      "2018-12-13 18:45:32.772 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 137: Model was successfully initialized!\n",
      "Model summary:\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, None, 300)    0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, None, 256)    230656      input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_2 (Conv1D)               (None, None, 256)    384256      input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, None, 256)    537856      input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, None, 256)    1024        conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, None, 256)    1024        conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, None, 256)    1024        conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, None, 256)    0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_2 (Activation)       (None, None, 256)    0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_3 (Activation)       (None, None, 256)    0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 256)          0           activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 256)          0           activation_2[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_3 (GlobalM (None, 256)          0           activation_3[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 768)          0           global_max_pooling1d_1[0][0]     \n",
      "                                                                 global_max_pooling1d_2[0][0]     \n",
      "                                                                 global_max_pooling1d_3[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 768)          0           concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 100)          76900       dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 100)          400         dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_4 (Activation)       (None, 100)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 100)          0           activation_4[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 7)            707         dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 7)            28          dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_5 (Activation)       (None, 7)            0           batch_normalization_5[0][0]      \n",
      "==================================================================================================\n",
      "Total params: 1,233,875\n",
      "Trainable params: 1,232,125\n",
      "Non-trainable params: 1,750\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# now one can initialize model\n",
    "m = build_model(config_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['GetWeather']]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m([\"Is it freezing in Offerman, California?\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's free memory\n",
    "del m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2018-12-13 18:45:33.675 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 97: Cannot find /home/dilyara/.deeppavlov/downloads/snips/valid.csv file\n",
      "2018-12-13 18:45:33.675 WARNING in 'deeppavlov.dataset_readers.basic_classification_reader'['basic_classification_reader'] at line 97: Cannot find /home/dilyara/.deeppavlov/downloads/snips/test.csv file\n",
      "2018-12-13 18:45:33.676 INFO in 'deeppavlov.dataset_iterators.basic_classification_iterator'['basic_classification_iterator'] at line 73: Splitting field <<train>> to new fields <<['train', 'valid']>>\n",
      "2018-12-13 18:45:33.679 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 100: [loading vocabulary from /home/dilyara/.deeppavlov/models/classifiers/intents_snips_v10/classes.dict]\n",
      "2018-12-13 18:45:33.680 INFO in 'deeppavlov.models.embedders.fasttext_embedder'['fasttext_embedder'] at line 52: [loading fastText embeddings from `/home/dilyara/.deeppavlov/downloads/embeddings/wiki.en.bin`]\n",
      "2018-12-13 18:45:54.568 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 287: [initializing `KerasClassificationModel` from saved]\n",
      "2018-12-13 18:45:54.913 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 297: [loading weights from model.h5]\n",
      "2018-12-13 18:45:55.112 INFO in 'deeppavlov.models.classifiers.keras_classification_model'['keras_classification_model'] at line 137: Model was successfully initialized!\n",
      "Model summary:\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, None, 300)    0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, None, 256)    230656      input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_2 (Conv1D)               (None, None, 256)    384256      input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, None, 256)    537856      input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, None, 256)    1024        conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, None, 256)    1024        conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, None, 256)    1024        conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, None, 256)    0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_2 (Activation)       (None, None, 256)    0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "activation_3 (Activation)       (None, None, 256)    0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 256)          0           activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_2 (GlobalM (None, 256)          0           activation_2[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_3 (GlobalM (None, 256)          0           activation_3[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 768)          0           global_max_pooling1d_1[0][0]     \n",
      "                                                                 global_max_pooling1d_2[0][0]     \n",
      "                                                                 global_max_pooling1d_3[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 768)          0           concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 100)          76900       dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 100)          400         dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_4 (Activation)       (None, 100)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 100)          0           activation_4[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 7)            707         dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 7)            28          dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_5 (Activation)       (None, 7)            0           batch_normalization_5[0][0]      \n",
      "==================================================================================================\n",
      "Total params: 1,233,875\n",
      "Trainable params: 1,232,125\n",
      "Non-trainable params: 1,750\n",
      "__________________________________________________________________________________________________\n",
      "2018-12-13 18:45:55.113 INFO in 'deeppavlov.core.commands.train'['train'] at line 207: Testing the best saved model\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"valid\": {\"eval_examples_count\": 1589, \"metrics\": {\"sets_accuracy\": 0.9824, \"f1_macro\": 0.982, \"roc_auc\": 0.9986}, \"time_spent\": \"0:00:01\"}}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'valid': OrderedDict([('sets_accuracy', 0.9824),\n",
       "              ('f1_macro', 0.982),\n",
       "              ('roc_auc', 0.9986)])}"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or one can evaluate model WITHOUT training\n",
    "evaluate_model(config_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow_kernel2",
   "language": "python",
   "name": "tensorflow_kernel2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}