VAE/09-VAE-with-CelebA-post.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n",
    "\n",
    "# <!-- TITLE --> [VAE9] - Data generation from latent space\n",
    "<!-- DESC --> Episode 5 : Exploring latent space to generate new data\n",
    "<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->\n",
    "\n",
    "## Objectives :\n",
    " - New data generation from **latent space**\n",
    " - Understanding of underlying principles\n",
    " - Guided image generation, **latent morphing**\n",
    " - Model management\n",
    " \n",
    "Here again, we don't consume data anymore, but we generate them ! ;-)\n",
    "\n",
    "\n",
    "The [CelebFaces Attributes Dataset (CelebA)](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) contains about 200,000 images (202599,218,178,3)...  \n",
    "...But our data is now in the imagination of our network!\n",
    "\n",
    "## What we're going to do :\n",
    " - Load a saved model\n",
    " - Reconstruct some images from latent space\n",
    " - Matrix of generated images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Init python stuff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.stats\n",
    "from skimage import io, transform\n",
    "import os,sys,importlib\n",
    "import math\n",
    "from importlib import reload\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from scipy.stats import norm\n",
    "\n",
    "from modules.datagen import DataGenerator\n",
    "from modules.models  import VAE\n",
    "\n",
    "sys.path.append('..')\n",
    "import fidle.pwk as pwk\n",
    "\n",
    "run_dir = './run/VAE8.001'\n",
    "datasets_dir = pwk.init('VAE9', run_dir)\n",
    "\n",
    "VAE.about()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Parameters\n",
    "**Note :** We only have one set of data, used for training.  \n",
    "We did not separate our data between learning and testing because our goal is to generate data.\n",
    "\n",
    "Define these parameters according to the clustered dataset you wish to use...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- Tests\n",
    "#\n",
    "image_size   = (128,128)\n",
    "enhanced_dir = './data'\n",
    "\n",
    "# --- Full clusters (128,128)\n",
    "#\n",
    "# image_size   = (128,128)\n",
    "# enhanced_dir = f'{datasets_dir}/celeba/enhanced'\n",
    "\n",
    "# ---- Full clusters (192,160)\n",
    "#\n",
    "# image_size   = (192,160)\n",
    "# enhanced_dir = f'{datasets_dir}/celeba/enhanced'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ---- Used for continous integration - Just forget this line\n",
    "#\n",
    "pwk.override('image_size', 'enhanced_dir')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Gets some data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ---- the place of the clusters files\n",
    "\n",
    "lx,ly        = image_size\n",
    "train_dir    = f'{enhanced_dir}/clusters-{lx}x{ly}'\n",
    "dataset_csv  = f'{datasets_dir}/celeba/origine/list_attr_celeba.csv'\n",
    "dataset_img  = f'{datasets_dir}/celeba/origine/img_align_celeba'\n",
    "\n",
    "# ---- Get images (one cluster)\n",
    "\n",
    "x_data       = np.load(f'{train_dir}/images-000.npy')\n",
    "\n",
    "# ---- Get descriptions\n",
    "\n",
    "dataset_desc = pd.read_csv(dataset_csv, header=0)\n",
    "\n",
    "print('Data directory is :',train_dir)\n",
    "print('Images retrieved  :',len(x_data))\n",
    "print('Descriptions      :',len(dataset_desc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Reload best model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vae=VAE()\n",
    "vae.reload(f'{run_dir}/models/best_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 - Image reconstruction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_show = 8\n",
    "np.random.shuffle(x_data)\n",
    "\n",
    "# ---- Get latent points and reconstructed images\n",
    "\n",
    "# y_reconst = vae.predict(x_data)\n",
    "\n",
    "z_mean, z_log_var, z_data    = vae.encoder.predict(x_data)\n",
    "y_reconst                    = vae.decoder.predict(z_data)\n",
    "\n",
    "# ---- Just show it\n",
    "\n",
    "pwk.plot_images(x_data[:10],    None, columns=10, x_size=1.5,y_size=1.5, spines_alpha=0.1, save_as='01-original')\n",
    "pwk.plot_images(y_reconst[:10], None, columns=10, x_size=1.5,y_size=1.5, spines_alpha=0.1, save_as='02-reconstruct')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6 - Latent space distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z_dim = z_data.shape[1]\n",
    "x = np.linspace(-3, 3, 100)\n",
    "\n",
    "fig = plt.figure(figsize=(12, 10))\n",
    "fig.subplots_adjust(hspace=0.3, wspace=0.2)\n",
    "\n",
    "for i in range(40):\n",
    "    ax = fig.add_subplot(4, 10, i+1)\n",
    "    ax.hist(z_data[:,i], density=True, bins = 20)\n",
    "    ax.axis('off')\n",
    "    ax.set_xlim(-3,3)\n",
    "    ax.text(0.5, -0.2, str(i), fontsize=14, ha='center', transform=ax.transAxes)\n",
    "    ax.plot(x,norm.pdf(x))\n",
    "\n",
    "pwk.save_fig('03-latent-space')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7 - Generation of new faces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_new = 48\n",
    "\n",
    "z_new = np.random.normal( loc=0,scale=0.7,size=(n_new,z_dim) )\n",
    "x_new = vae.decoder.predict(z_new)\n",
    "\n",
    "pwk.plot_images(x_new, None, columns=6, x_size=2,y_size=2.4, spines_alpha=0,y_padding=0, save_as='04-new-faces')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8 - Playing with latent space\n",
    "### 8.1 - The attributes of our images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwk.subtitle('Dataset description file (csv) :')\n",
    "display(dataset_desc.head())\n",
    "\n",
    "pwk.subtitle('Defined attributes :')\n",
    "for i,v in enumerate(dataset_desc.columns):\n",
    "    print(f'{v:24}', end='')\n",
    "    if (i+1) % 4 == 0 :print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.2 Let's find some predictable images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_latent_vector(images_desc, vector_size=50):\n",
    "    \"\"\"\n",
    "    Get a set of images, give them to the encoder and return an mean vector\n",
    "    args:\n",
    "        images_desc : Images descrption\n",
    "    return:\n",
    "        mean(z)\n",
    "    \"\"\"\n",
    "\n",
    "    # ---- Get filenames of given images descriptions\n",
    "    \n",
    "    filenames=images_desc['image_id'][:vector_size]\n",
    "    \n",
    "    # ---- Retrieve images\n",
    "    \n",
    "    imgs=[]\n",
    "    print(f'Read {vector_size} images...', end='')\n",
    "    for i,filename in enumerate(filenames):\n",
    "        filename = f'{dataset_img}/{filename}'\n",
    "        img = io.imread(filename)\n",
    "        img = transform.resize(img, image_size)\n",
    "        imgs.append( img )\n",
    "    print('done.')\n",
    "            \n",
    "    # ---- Get latent space vectors\n",
    "\n",
    "    x_images=np.array(imgs)\n",
    "    z_mean, z_log_var, z  = vae.encoder.predict(x_images)\n",
    "    \n",
    "    # ---- return mean vector\n",
    "    \n",
    "    return z.mean(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = dataset_desc\n",
    "\n",
    "z11 = get_latent_vector( df.loc[ (df['Male'] == -1)  & (df['Smiling']== 1) & (df['Blond_Hair']== 1)] )\n",
    "z12 = get_latent_vector( df.loc[ (df['Male'] == -1)  & (df['Smiling']== 1) & (df['Black_Hair']== 1)] )\n",
    "z21 = get_latent_vector( df.loc[ (df['Male'] ==  1)  & (df['Smiling']==-1) & (df['Black_Hair']== 1)] )\n",
    "\n",
    "labels=['Woman\\nBlond hair\\nSmiling','Woman\\nBlack hair\\nSmiling','Man\\nBlack Hair\\nNot smiling']\n",
    "\n",
    "\n",
    "z_images = np.array( [z11,z12,z21] )\n",
    "x_images = vae.decoder.predict( z_images, verbose=0 )\n",
    "pwk.plot_images(x_images,labels,columns=3,x_size=3,y_size=3,spines_alpha=0, save_as='05-predictable')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.3 - And do somme latent morphing !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "n=6\n",
    "dj=(z12-z11)/n\n",
    "di=(z21-z11)/n\n",
    "\n",
    "z=[]\n",
    "for i in range(n):\n",
    "    for j in range(n):\n",
    "        z.append( z11+di*i+dj*j )\n",
    "\n",
    "x_images = vae.decoder.predict( np.array(z) )\n",
    "pwk.plot_images(x_images,columns=n,x_size=2,y_size=2.4,y_padding=0,spines_alpha=0, save_as='06-morphing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwk.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<img width=\"80px\" src=\"../fidle/img/00-Fidle-logo-01.svg\"></img>"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "8e38643e33497db9a306e3f311fa98cb1e65371278ca73ee4ea0c76aa5a4f387"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit ('fidle-cpu': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}