莎士比亚风格文本生成.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "说明：此生成模型是基于字符的。训练一个模型预测该序列的下一个字符，重复调用该模型，从而生成更长的文本序列。\n",
    "\n",
    "虽然有些句子符合语法规则，但是大多数句子没有意义。这个模型尚未学习到单词的含义，\n",
    "\n",
    "此模型是基于字符的。训练开始时，模型不知道如何拼写一个英文单词，甚至不知道单词是文本的一个单位。\n",
    "输出文本的结构类似于剧本 -- 文本块通常以讲话者的名字开始；而且与数据集类似，讲话者的名字采用全大写字母。\n",
    "如下文所示，此模型由小批次 （batch） 文本训练而成（每批 100 个字符）。即便如此，此模型仍然能生成更长的文本序列，并且结构连贯。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import os\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_file = tf.keras.utils.get_file('shakespeare.txt', 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length of text:1115394 characters.\n"
     ]
    }
   ],
   "source": [
    "# 读取数据并解码\n",
    "text=open(path_to_file,'rb').read().decode(encoding='utf-8')\n",
    "print(\"length of text:{} characters.\".format(len(text)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n",
      "All:\n",
      "Resolved. resolved.\n",
      "\n",
      "First Citizen:\n",
      "First, you know Caius Marcius is chief enemy to the people.\n",
      "\n",
      "All:\n",
      "We know't, we know't.\n",
      "\n",
      "First Citizen:\n",
      "Let us kill him, and we'll have corn at our own price.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(text[:349])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "unique characters:65\n"
     ]
    }
   ],
   "source": [
    "# 查看文章中的字符\n",
    "vocab=sorted(set(text))\n",
    "print('unique characters:{}'.format(len(vocab)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理文本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 向量化文本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们首先要将字符串映射到数字表示，创建两个字典\n",
    "一个用于查找字符映射到数字，一个用于将数字映射到字符"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\n', ' ', '!', '$', '&', \"'\", ',', '-', '.', '3', ':', ';', '?', 'A', 'B']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1115394"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 字符到数字\n",
    "char2idx={u:i for (i,u) in enumerate(vocab)}\n",
    "# 数字映射到字符\n",
    "idx2char=np.array(vocab)\n",
    "# 将文本转换为数字\n",
    "text_as_int=np.array([char2idx[c] for c in text])\n",
    "len(text_as_int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<zip at 0x6466db5c8>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# zip() 函数用于将可迭代的对象作为参数，将对象中对应的元素打包成一个个元组，然后返回由这些元组组成的列表。\n",
    "a = [1,2,3]\n",
    "b = [4,5,6]\n",
    "c = [4,5,6,7,8]\n",
    "zipped = zip(a,b)     # 打包为元组的列表\n",
    "# [(1, 4), (2, 5), (3, 6)]\n",
    "zip(a,c)              # 元素个数与最短的列表一致\n",
    "# [(1, 4), (2, 5), (3, 6)]\n",
    "zip(*zipped)          # 与 zip 相反，*zipped 可理解为解压，返回二维矩阵式\n",
    "# [(1, 2, 3), (4, 5, 6)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "'\\n':   0,\n",
      "' ' :   1,\n",
      "'!' :   2,\n",
      "'$' :   3,\n",
      "'&' :   4,\n",
      "\"'\" :   5,\n",
      "',' :   6,\n",
      "'-' :   7,\n",
      "'.' :   8,\n",
      "'3' :   9,\n",
      "':' :  10,\n",
      "';' :  11,\n",
      "'?' :  12,\n",
      "'A' :  13,\n",
      "'B' :  14,\n",
      "'C' :  15,\n",
      "'D' :  16,\n",
      "'E' :  17,\n",
      "'F' :  18,\n",
      "'G' :  19,\n",
      "...}\n"
     ]
    }
   ],
   "source": [
    "# 查看前20个字符映射到数字\n",
    "print('{')\n",
    "for char,_ in zip(char2idx,range(20)):\n",
    "    #repr() 函数将对象转化为供解释器读取的形式。\n",
    "    print(\"{:4s}: {:3d},\".format(repr(char),char2idx[char]))\n",
    "print('...}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Citizen --- mapping to num --->[18, 47, 56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52]\n"
     ]
    }
   ],
   "source": [
    "# 显示文本前13个字符映射到数字\n",
    "print(f'{text[:13]} --- mapping to num --->{[char2idx[c] for c in text[:13]]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "文件读取完成之后，我们需要明白我们的任务。\n",
    "我们读取一个字符，然后预测下一个最有可能的字符。\n",
    "我们需要训练一个模型用于预测每一个时间步最有可能的输出"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建训练样本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将文本切分为样本序列，每个输入序列包含文本的seq_lebgth个字符\n",
    "对于每一个输入序列，其目标序列包含对应长度相同的文本，但是向右移一个字符。\n",
    "\n",
    "将文本拆分为长度为seq_length+1的文本块。假设seq_length=4,文本为‘hello’。那么输入序列为‘Hell',目标序列为'ello'。\n",
    "首先使用tf.data.Dataset.from_tensor_slices将文本向量转为字符索引流"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(18, shape=(), dtype=int64)\n",
      "F\n",
      "tf.Tensor(47, shape=(), dtype=int64)\n",
      "i\n",
      "tf.Tensor(56, shape=(), dtype=int64)\n",
      "r\n",
      "tf.Tensor(57, shape=(), dtype=int64)\n",
      "s\n",
      "tf.Tensor(58, shape=(), dtype=int64)\n",
      "t\n"
     ]
    }
   ],
   "source": [
    "# 设定句子长度\n",
    "seq_length=100\n",
    "example_per_epoch=len(text)//seq_length\n",
    "\n",
    "# 创建训练样本/目标\n",
    "char_dataset=tf.data.Dataset.from_tensor_slices(text_as_int)\n",
    "\n",
    "for i in char_dataset.take(5):\n",
    "    print(i)\n",
    "    print(idx2char[i.numpy()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[18 47 56 57 58  1 15 47 58 47 64 43 52 10  0 14 43 44 53 56 43  1 61 43\n",
      "  1 54 56 53 41 43 43 42  1 39 52 63  1 44 59 56 58 46 43 56  6  1 46 43\n",
      " 39 56  1 51 43  1 57 54 43 39 49  8  0  0 13 50 50 10  0 31 54 43 39 49\n",
      "  6  1 57 54 43 39 49  8  0  0 18 47 56 57 58  1 15 47 58 47 64 43 52 10\n",
      "  0 37 53 59  1], shape=(101,), dtype=int64)\n",
      "'First Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou '\n",
      "tf.Tensor(\n",
      "[39 56 43  1 39 50 50  1 56 43 57 53 50 60 43 42  1 56 39 58 46 43 56  1\n",
      " 58 53  1 42 47 43  1 58 46 39 52  1 58 53  1 44 39 51 47 57 46 12  0  0\n",
      " 13 50 50 10  0 30 43 57 53 50 60 43 42  8  1 56 43 57 53 50 60 43 42  8\n",
      "  0  0 18 47 56 57 58  1 15 47 58 47 64 43 52 10  0 18 47 56 57 58  6  1\n",
      " 63 53 59  1 49], shape=(101,), dtype=int64)\n",
      "'are all resolved rather to die than to famish?\\n\\nAll:\\nResolved. resolved.\\n\\nFirst Citizen:\\nFirst, you k'\n"
     ]
    }
   ],
   "source": [
    "# 我们可以使用batch方法，将单个字符转换为所需要的长度序列\n",
    "# 注意不是pad_batch\n",
    "# 因为dataset中每一个字符就是一个样本，dataset是一个总长为1115394\n",
    "# 这就表示我们有1115393个输入字符\n",
    "# 有1115393个目标字符\n",
    "# 我们使用batch方法，将每一条序列长度变为101，这就是我们一条原始的文本\n",
    "# 之后我们再将其变为输入文本与输出文本\n",
    "\n",
    "# 注意每条序列长度是seq_length+1\n",
    "# 输入长度是seq_length,目标长度是seq_length\n",
    "sequences=char_dataset.batch(seq_length+1,drop_remainder=True)\n",
    "\n",
    "for item in sequences.take(2):\n",
    "    print(item)\n",
    "    print(repr(''.join(idx2char[item.numpy()])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来要创建训练样本与输出目标样本\n",
    "对于每一个序列，我们使用map方法先复制，在顺移一位，从而创建输入与目标\n",
    "map函数可以将一个函数应用到每一个batch上"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_input_target(chunk):\n",
    "    input_text=chunk[:-1]\n",
    "    target_text=chunk[1:]\n",
    "    return input_text,target_text\n",
    "dataset=sequences.map(split_input_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<MapDataset shapes: ((100,), (100,)), types: (tf.int64, tf.int64)>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# dataset中包含了输入与目标文本\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input is: 'First Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou'\n",
      "target is: 'irst Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou '\n"
     ]
    }
   ],
   "source": [
    "for input_example,target_example in dataset.take(1):\n",
    "    print('input is: {}'.format(repr(''.join(idx2char[input_example.numpy()]))))\n",
    "    print('target is: {}'.format(repr(''.join(idx2char[target_example.numpy()]))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在每一个时间步，会从input中接受一个字符，然后预测目标是target中对应索引的字符。如输入'F'，尝试预测‘i’。在下一个时间步，接受‘i’，尝试预测‘r’。使用RNN，不仅会考虑当前时间步的输入，也会考虑之前时间步的输入。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "timestep 0:\n",
      "  the input is :F\n",
      "  the target is:i\n",
      "timestep 1:\n",
      "  the input is :i\n",
      "  the target is:r\n",
      "timestep 2:\n",
      "  the input is :r\n",
      "  the target is:s\n",
      "timestep 3:\n",
      "  the input is :s\n",
      "  the target is:t\n",
      "timestep 4:\n",
      "  the input is :t\n",
      "  the target is: \n"
     ]
    }
   ],
   "source": [
    "for i,(input_idx,target_idx) in enumerate((zip(input_example[:5],target_example[:5]))):\n",
    "    print(f'timestep {i}:')\n",
    "    print(f'  the input is :{idx2char[input_idx]}')\n",
    "    print(f'  the target is:{idx2char[target_idx]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建训练批次"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BatchDataset shapes: ((64, 100), (64, 100)), types: (tf.int64, tf.int64)>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "BATCH_SIZE=64\n",
    "\n",
    "# 设置缓冲区大小，以重新排列数据集\n",
    "# （TF 数据被设计为可以处理可能是无限的序列，\n",
    "# 所以它不会试图在内存中重新排列整个序列。相反，\n",
    "# 它维持一个缓冲区，在缓冲区重新排列元素。） \n",
    "BUFFER_SIZE=10000\n",
    "\n",
    "dataset=dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE,drop_remainder=True)\n",
    "# 现在dataset每条输入样本长为100，一批样本为64条\n",
    "# dataset=dataset.batch(BATCH_SIZE,drop_remainder=True)\n",
    "\n",
    "\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_batch=[]\n",
    "for i in dataset.take(3):\n",
    "    text_batch.append(i[0].numpy())\n",
    "    text_batch.append(i[1].numpy())\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n defending of myself,\n",
      "A traitor to my God, my king, and me:\n",
      "And as I truly fight, defend me heaven!\n",
      "---\n",
      "ank thee, gentle Percy; and be sure\n",
      "I count myself in nothing else so happy\n",
      "As in a soul remembering\n"
     ]
    }
   ],
   "source": [
    "for i in range(1):\n",
    "    print(''.join(idx2char[text_batch[0][0]]))\n",
    "    print('---')\n",
    "    print(''.join(idx2char[text_batch[0][1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size=len(vocab)\n",
    "embedding_dim=256\n",
    "rnn_units=1024\n",
    "batch_size=BATCH_SIZE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于为什么要在embedding层加入batch_input_shape  \n",
    "ValueError: If a RNN is stateful, it needs to know its batch size. Specify the batch size of your input tensors: \n",
    "- If using a Sequential model, specify the batch size by passing a `batch_input_shape` argument to your first layer.\n",
    "- If using the functional API, specify the batch size by passing a `batch_shape` argument to your Input layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是一些关于GRU的参数说明"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.\n",
    "即会将上一个批次的最后一个状态作为下一个批次的初始状态"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence. Default: False  \n",
    "是返回输出序列中的最后一个输出，还是返回完整序列\n",
    "==true输出全部，false输出最后一个\n",
    "控制的是hidden_state\n",
    "\n",
    "注意区别return_state:控制的是cell_state.\n",
    "return_state=true输出，false不输出"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处有一个疑问，使用了stateful=true  \n",
    "但是数据集连续的batch之间同一个index i 处的数据我觉得并没有连续"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(vocab_size, embedding_dim, rnn_units, batch_size):\n",
    "    model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Embedding(vocab_size, embedding_dim,\n",
    "                              batch_input_shape=[batch_size, None]),\n",
    "    tf.keras.layers.GRU(rnn_units,\n",
    "                        return_sequences=True,\n",
    "                        stateful=True,\n",
    "                        recurrent_initializer='glorot_uniform'),\n",
    "    tf.keras.layers.Dense(vocab_size,activation='softmax')\n",
    "  ])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = build_model(\n",
    "  vocab_size = len(vocab),\n",
    "  embedding_dim=embedding_dim,\n",
    "  rnn_units=rnn_units,\n",
    "  batch_size=BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image.png](./pic/text_generate1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "关于logits的理解\n",
    "也可以这么理解：logits与 softmax都属于在输出层的内容  \n",
    "logits = tf.matmul(X, W) + bias  \n",
    "再对logits做归一化处理，就用到了softma:  \n",
    "Y_pred = tf.nn.softmax(logits,name='Y_pred')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](pic/text_generation2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64, 100, 65)\n"
     ]
    }
   ],
   "source": [
    "# 检查模型输出形状\n",
    "for input_example_batch,target_example_batch in dataset.take(1):\n",
    "    example_batch_predictions=model(input_example_batch)\n",
    "    print(example_batch_predictions.shape) #(batch_size,seq_length,vocab_size)b\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(1.0000001, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(sum(example_batch_predictions[0][2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的例子虽然长度为100，但是我们的模型可以处理任何长度，因为在embedding层的batch_input_shape设置为了[batch_size,None],在model.summary()中查看"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (64, None, 256)           16640     \n",
      "_________________________________________________________________\n",
      "gru (GRU)                    (64, None, 1024)          3938304   \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (64, None, 65)            66625     \n",
      "=================================================================\n",
      "Total params: 4,021,569\n",
      "Trainable params: 4,021,569\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了进行预测，我们还要对模型的输出进行抽样（sample），从而获得输出的字符。\n",
    "这个分布是更具字符集的逻辑回归进行定义的  \n",
    "需要注意，进行sample是必要的，否则的话取分布的最大值的索引，这样模型就很有可能卡在循环中。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tips：\n",
    "使用tf.random.categorical从一个分类分布中抽取样本\n",
    "```\n",
    "tf.random.categorical(\n",
    "    logits,\n",
    "    num_samples,\n",
    "    dtype=None,\n",
    "    seed=None,\n",
    "    name=None\n",
    ")\n",
    "```\n",
    "- logits: 形状为 [batch_size, num_classes]的张量. 每个切片 [i, :] 代表对于所有类的未正规化的log概率。\n",
    "- num_samples: 0维，从每一行切片中抽取的独立样本的数量。  \n",
    "返回是样本的索引\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 6, 39,  6, 40, 24, 29, 49, 59, 27, 42, 44,  2,  6, 10, 20, 29, 27,\n",
       "        3,  0,  5, 37,  2, 10, 47, 62, 36, 40, 31, 13, 18, 47, 13, 46, 59,\n",
       "        3, 42, 61, 52, 57,  7, 50, 25, 58, 28, 25, 41,  5, 29, 32, 49, 33,\n",
       "       27, 49, 39,  9, 33, 36, 57,  4, 17,  9, 37, 22,  7, 48, 64, 21, 27,\n",
       "       28, 25, 61, 12, 17, 30, 20, 60, 11, 49, 24, 35, 48, 39, 41, 63,  8,\n",
       "        7, 34, 59, 45, 44, 14,  8, 59, 38, 63, 19, 20, 56, 45, 43])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_indeces=tf.random.categorical(example_batch_predictions[0],num_samples=1)\n",
    "sample_indeces=tf.squeeze(sample_indeces,axis=-1).numpy() # tf.squeeze删除一个维度\n",
    "# 输出sample_indeces,即为我们依据分布进行抽样得到的下一个预测字符的索引\n",
    "sample_indeces\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input data:'ajesty\\nWith those gross taunts I often have endured.\\nI had rather be a country servant-maid\\nThan a g'\n",
      "......\n",
      "prediction without training:\",a,bLQkuOdf!,:HQO$\\n'Y!:ixXbSAFiAhu$dwns-lMtPMc'QTkUOka3UXs&E3YJ-jzIOPMw?ERHv;kLWjacy.-VugfB.uZyGHrge\"\n"
     ]
    }
   ],
   "source": [
    "# 将索引转换为字符，查看未训练之前所得到的输出\n",
    "print('input data:{}'.format(repr(''.join(idx2char[input_example_batch[0]]))))\n",
    "print('......')\n",
    "print('prediction without training:{}'.format(repr(''.join(idx2char[sample_indeces]))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，为训练之前，我们的模型输出是杂乱无章的，不知道单词的组成，文本的格式等等，跟不用说语法与语义"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们的模型可以看作一个多分类问题，最后预测下一个字符的类别  \n",
    "\n",
    "\n",
    "#### 一个疑问，为什么最后一层不使用softmax激活函数，这样最后就不需要对输出进行逻辑回归将其转化为概率进行抽样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "example mean loss:4.174365997314453\n"
     ]
    }
   ],
   "source": [
    "# 由于模型返回的是逻辑回归，所以我们需要设定参数from_logits\n",
    "def loss(labels,logits):\n",
    "    return tf.keras.losses.sparse_categorical_crossentropy(labels,logits,from_logits=True)\n",
    "exapmle_batch_loss=loss(target_example_batch,example_batch_predictions)\n",
    "print(f'example mean loss:{exapmle_batch_loss.numpy().mean()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',loss=loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------load the model-----------------\n"
     ]
    }
   ],
   "source": [
    "checkpoint_save_path = \"./text_generation_checkpoint/text_generation.ckpt\"\n",
    "\n",
    "if os.path.exists(checkpoint_save_path + '.index'):\n",
    "    print('-------------load the model-----------------')\n",
    "    model.load_weights(checkpoint_save_path)\n",
    "\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_save_path,\n",
    "                                                 save_weights_only=True,\n",
    "                                                 # monitor='loss',\n",
    "                                                 # save_best_only=True,\n",
    "                                                 verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 已经在colab上训练了30个epoch\n",
    "history = model.fit(dataset, epochs=EPOCHS, callbacks=[cp_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成文本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 回复检查点"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将批大小设置为1，由于之前训练的时候设定的批大小为64，我们要使用不同的batch_size。我们需要重新建立模型，从checkpoint恢复权重即可"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------load the model-----------------\n",
      "./text_generation_checkpoint/text_generation.ckpt\n"
     ]
    }
   ],
   "source": [
    "model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)\n",
    "\n",
    "checkpoint_save_path = \"./text_generation_checkpoint/text_generation.ckpt\"\n",
    "\n",
    "# 加载训练好的模型，本地训练太慢了，在colab中训练完毕了\n",
    "if os.path.exists(checkpoint_save_path + '.index'):\n",
    "    print('-------------load the model-----------------')\n",
    "    print(checkpoint_save_path)\n",
    "    model.load_weights(checkpoint_save_path)\n",
    "\n",
    "# choose to manually build your model by calling `build(batch_input_shape)`:\n",
    "model.build(tf.TensorShape([1, None]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (1, None, 256)            16640     \n",
      "_________________________________________________________________\n",
      "gru_1 (GRU)                  (1, None, 1024)           3938304   \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (1, None, 65)             66625     \n",
      "=================================================================\n",
      "Total params: 4,021,569\n",
      "Trainable params: 4,021,569\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 循环预测"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 首先我们需要设置起始字符串，初始化RNN的状态，然后设置生成字符的个数\n",
    "- 然后利用起始字符串和RNN的状态，获取下一个字符的预测分布\n",
    "- 然后使用分类分布计算预测字符串的索引，然后将这个字符串当作模型下一个时间步的输入\n",
    "- 模型返回的 RNN 状态被输送回模型。现在，模型有更多上下文可以学习，而非只有一个字符。在预测出下一个字符后，更改过的 RNN 状态被再次输送回模型。模型就是这样，通过不断从前面预测的字符获得更多上下文，进行学习。  \n",
    "\n",
    "![](pic/text_generation3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之后我们会发现，我们的模型已经能够写出正确格式的莎士比亚风格诗句，知道什么时候大写，什么时候空格，什么时候分段。当然，对于语法和语义还没有学习到什么"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_text(model,start_string):\n",
    "    num_generate=1000 #生成字符数\n",
    "    \n",
    "    # 将起始字符转换为数字\n",
    "    input_eval=[char2idx[s] for s in start_string]\n",
    "    # 增加一个维度，并且可将输入变为张量\n",
    "    input_eval=tf.expand_dims(input_eval,0)\n",
    "    \n",
    "    # 存储结果\n",
    "    text_generated=[]\n",
    "    \n",
    "    # 低温度会生成更可预测的文本\n",
    "    # 较高温度会生成更令人惊讶的文本\n",
    "    # 可以通过试验以找到最好的设定\n",
    "    \n",
    "    # 更高的温度得到的是熵更大的采样分布，会生成更加出人意料、更加无结构的生成数据，\n",
    "    # 而更低的温度对应更小的随机性，以及更加可预测的生成数据。\n",
    "    temperature = 1.0\n",
    "    \n",
    "    model.reset_states()\n",
    "    \n",
    "    for i in range(num_generate):\n",
    "        \n",
    "        # 此时shape是 [batch_size=1,seq_length,voacb_size]\n",
    "        predictions=model(input_eval)\n",
    "        \n",
    "\n",
    "        # 此时shape是 [seq_length,voacb_size]\n",
    "        predictions=tf.squeeze(predictions,0)\n",
    "        \n",
    "#         pred=tf.keras.activations.softmax(predictions).numpy()\n",
    "#         print(pred.shape)\n",
    "        \n",
    "        \n",
    "        # 依据分布进行抽样\n",
    "        predictions=predictions/temperature\n",
    "        # tf.random.categorical返回的是一个二维的tensor\n",
    "        # shape=(batch_size,num_samples)\n",
    "        # [-1,0]即取返回值的最后一个batch_size的第一个元素\n",
    "        # 因为我们输入可能是多个字符，如‘ROME’，输出维度就是（4,vocab_size=65)\n",
    "        # 所以我们用[-1,0]来获得“ROME’中最后一个‘E’的下一个抽样产生的输出（sample）\n",
    "        prediction_index=tf.random.categorical(predictions,num_samples=1)[-1,0].numpy()\n",
    "\n",
    "        \n",
    "#         pred=np.array(pred)[-1,:]\n",
    "#         print(pred.shape)\n",
    "        # p代表每个元素选取的概率\n",
    "#         prediction_index = np.random.choice(list(range(65)), p=pred.ravel())\n",
    "        \n",
    "        # 将上一个预测的字符和之前的状态传入模型，作为下一个输入\n",
    "        input_eval=tf.expand_dims([prediction_index],0)\n",
    "        text_generated.append(idx2char[prediction_index])\n",
    "        \n",
    "    return start_string +''.join(text_generated)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[3]], shape=(1, 1), dtype=int64)\n",
      "tf.Tensor([[3]], shape=(1, 1), dtype=int64)\n",
      "tf.Tensor([[0]], shape=(1, 1), dtype=int64)\n",
      "tf.Tensor([[1]], shape=(1, 1), dtype=int64)\n",
      "tf.Tensor([[2]], shape=(1, 1), dtype=int64)\n",
      "tf.Tensor([[1]], shape=(1, 1), dtype=int64)\n",
      "tf.Tensor([[3]], shape=(1, 1), dtype=int64)\n",
      "tf.Tensor([[1]], shape=(1, 1), dtype=int64)\n",
      "tf.Tensor([[1]], shape=(1, 1), dtype=int64)\n",
      "tf.Tensor([[1]], shape=(1, 1), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "for i in tf.range(10):\n",
    "    samples = tf.random.categorical([[1.0,1.0,1.0,1.0,1.0]], 1)\n",
    "    print(samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROMEO: EbzKoBbGwvirscyPj!$GpwYMaQWlZ!\n",
      "g-n,ugA&JNb';vnuhwGl\n",
      "B:-zt&\n",
      "&h\n",
      "S-vXZGa!upktq3xsGPCSXHTY&GY'aF'feFc?dWhvARvIsxLX.BXPMcIbBbcMGNBNIjFj&3Mk-Y;QR-dZ\n",
      "l&YcVLVYRugMw,SSbn'xnGmaS3-gnzjNHS?B?QMYro;Z3&YPkyVPqFHvQKO!$3SNP!PSCpm$'eSzY; f!Q,WRl;d:'di:hTWtaAMmVyBNfhb!CdtE:EQS,isyCyXFd?-$yB$u&SK.,V;b;bkx;WGKJsuj:'tZMrqHJ,pjCkpk3XD&GSPzK;yCsdax.rLPeg,qqJ-P'fB.fT?oD-Oguxdhs&FLogDEayeSW;NDDZb;x!d;?z&kv'ffJwxCeR?I;XdYDyFlXrcYgSskdL vv-'$pQw\n",
      "-$FRmplNTK&pISyHpKFmXqItMNL!Ccs!RXlHP:&,.Ys\n",
      "3''?mcUw-WlSOEF\n",
      "'TaZ;VlRXicLyy3mWrDs\n",
      "vBTQgJkUb&MBblO,QMQF-;NOTlnTGdSGP$au;:jsEPgWCrA! ,xiTPYxrMzB&.lsSurDYc,WP!FP-EsTu.OHVzsyF&RxkI&-P daZWgxVleZU;JLxUh?chjMhlwFIJ.'&wDWbTLEp!,DQueh;sAM;qWpEeGa,fxcn!Hxzn3w,iy,GEwvJ&Pkr.I-CzJSUPf\n",
      "ZWSJeg;o'M;SLmtAgPfgSUsS b\n",
      "LX:&.hq'mDpf$;vEnS-M$rskRp?RseyNS,\n",
      "gKmboSqBkIj.RfutEMJcCCM,pSKDY.ZMNizbKQN'gFRUZVVWmX.Udy 3Oc;LOt&M'lZk'wmeL;YhBU;ezrNy,.A\n",
      ":PhmnmCT-iDTNv;f,k;.$FIjm Kq$AoIWcxrTqu:RtdL'IFvIMvBkKD3AKzU$fqEPHHlDMlgGqeqiB!tiCGrIvyWDkvgqLa!lfDn-w:lZMsIzARmkTFvJ&KISZG;DW&nywxH-OyrlspVeeZxm!eK$P.eR\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"\\nROMEO: I advance fiture each other,\\nHow many haughty love, your own suspicion from so rounder he divide,\\nAs if I had some all fell.\\n\\nFullow:\\nBleased the soldiers, Cleome,\\nAnd thou hadst beat me back to Man\\nIn an outward stars that sle with thee?\\nWhy should she noble endary?\\n\\nDUKE OF YORK:\\n'Twas something I have you aud in France,\\nAnd rear ourselves: 'tis he that lives in the substance where\\nThey are buts for a schollow.\\n\\nCAPULET:\\nGod and for all his own good will doth lack some general.\\n\\nGire descings beasts do go.\\n\\nLADY GREY:\\nMy lords, so amel, or ho! You are plack'd,\\nAnd nother ready straight\\nAnd ragers else to make in piece of my mind.\\n\\nWARWICK:\\nAy for my middless sin with arms:\\nBe you, covert:\\nWe cannot blow our needs, even whether I wear your highness\\nWill up my master read it in his high;\\nTo-morrow or perpetual speech, have you know the drowsy overworn:\\nWhen I would be the rest receive an offer;\\nWhy, why, your fearful souls thy head,\\nAnd errs as swiftly, sir;\\nHortensio after largers, fr\\n\\n\""
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(generate_text(model, start_string=u\"ROMEO: \"))\n",
    "# 很奇怪，权重都是保存在谷歌云盘上，下载下来的，在本地结果很糟糕\n",
    "# 在colab上加载相同的模型权重\n",
    "# 效果如下：\n",
    "'''\n",
    "ROMEO: I advance fiture each other,\n",
    "How many haughty love, your own suspicion from so rounder he divide,\n",
    "As if I had some all fell.\n",
    "\n",
    "Fullow:\n",
    "Bleased the soldiers, Cleome,\n",
    "And thou hadst beat me back to Man\n",
    "In an outward stars that sle with thee?\n",
    "Why should she noble endary?\n",
    "\n",
    "DUKE OF YORK:\n",
    "'Twas something I have you aud in France,\n",
    "And rear ourselves: 'tis he that lives in the substance where\n",
    "They are buts for a schollow.\n",
    "\n",
    "CAPULET:\n",
    "God and for all his own good will doth lack some general.\n",
    "\n",
    "Gire descings beasts do go.\n",
    "\n",
    "LADY GREY:\n",
    "My lords, so amel, or ho! You are plack'd,\n",
    "And nother ready straight\n",
    "And ragers else to make in piece of my mind.\n",
    "\n",
    "WARWICK:\n",
    "Ay for my middless sin with arms:\n",
    "Be you, covert:\n",
    "We cannot blow our needs, even whether I wear your highness\n",
    "Will up my master read it in his high;\n",
    "To-morrow or perpetual speech, have you know the drowsy overworn:\n",
    "When I would be the rest receive an offer;\n",
    "Why, why, your fearful souls thy head,\n",
    "And errs as swiftly, sir;\n",
    "Hortensio after largers, fr\n",
    "\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}