tfoptflow/pwcnet_train_sm-6-2-cyclic-chairsthingsmix.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PWC-Net-small model training (with cyclical learning rate schedule)\n",
    "=======================================================\n",
    "\n",
    "In this notebook we:\n",
    "- Use a small model (no dense or residual connections), 6 level pyramid, uspample level 2 by 4 as the final flow prediction\n",
    "- Train the PWC-Net-small model on a mix of the `FlyingChairs` and `FlyingThings3DHalfRes` dataset using a Cyclic<sub>short</sub> schedule of our own\n",
    "- The Cyclic<sub>short</sub> schedule oscillates between `5e-04` and `1e-05` for 200,000 steps\n",
    "\n",
    "Below, look for `TODO` references and customize this notebook based on your own needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "\n",
    "[2018a]<a name=\"2018a\"></a> Sun et al. 2018. PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume. [[arXiv]](https://arxiv.org/abs/1709.02371) [[web]](http://research.nvidia.com/publication/2018-02_PWC-Net%3A-CNNs-for) [[PyTorch (Official)]](https://github.com/NVlabs/PWC-Net/tree/master/PyTorch) [[Caffe (Official)]](https://github.com/NVlabs/PWC-Net/tree/master/Caffe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "pwcnet_train.ipynb\n",
    "\n",
    "PWC-Net model training.\n",
    "\n",
    "Written by Phil Ferriere\n",
    "\n",
    "Licensed under the MIT License (see LICENSE for details)\n",
    "\n",
    "Tensorboard:\n",
    "    [win] tensorboard --logdir=E:\\\\repos\\\\tf-optflow\\\\tfoptflow\\\\pwcnet-sm-6-2-cyclic-chairsthingsmix\n",
    "    [ubu] tensorboard --logdir=/media/EDrive/repos/tf-optflow/tfoptflow/pwcnet-sm-6-2-cyclic-chairsthingsmix\n",
    "\"\"\"\n",
    "from __future__ import absolute_import, division, print_function\n",
    "import sys\n",
    "from copy import deepcopy\n",
    "\n",
    "from dataset_base import _DEFAULT_DS_TRAIN_OPTIONS\n",
    "from dataset_flyingchairs import FlyingChairsDataset\n",
    "from dataset_flyingthings3d import FlyingThings3DHalfResDataset\n",
    "from dataset_mixer import MixedDataset\n",
    "from model_pwcnet import ModelPWCNet, _DEFAULT_PWCNET_TRAIN_OPTIONS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO: Set this first!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: You MUST set dataset_root to the correct path on your machine!\n",
    "if sys.platform.startswith(\"win\"):\n",
    "    _DATASET_ROOT = 'E:/datasets/'\n",
    "else:\n",
    "    _DATASET_ROOT = '/media/EDrive/datasets/'\n",
    "_FLYINGCHAIRS_ROOT = _DATASET_ROOT + 'FlyingChairs_release'\n",
    "_FLYINGTHINGS3DHALFRES_ROOT = _DATASET_ROOT + 'FlyingThings3D_HalfRes'\n",
    "    \n",
    "# TODO: You MUST adjust the settings below based on the number of GPU(s) used for training\n",
    "# Set controller device and devices\n",
    "# A one-gpu setup would be something like controller='/device:GPU:0' and gpu_devices=['/device:GPU:0']\n",
    "# Here, we use a dual-GPU setup, as shown below\n",
    "gpu_devices = ['/device:GPU:0', '/device:GPU:1']\n",
    "controller = '/device:CPU:0'\n",
    "\n",
    "# TODO: You MUST adjust this setting below based on the amount of memory on your GPU(s)\n",
    "# Batch size\n",
    "batch_size = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pre-train on `FlyingChairs+FlyingThings3DHalfRes` mix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: You MUST set the batch size based on the capabilities of your GPU(s) \n",
    "#  Load train dataset\n",
    "ds_opts = deepcopy(_DEFAULT_DS_TRAIN_OPTIONS)\n",
    "ds_opts['in_memory'] = False                          # Too many samples to keep in memory at once, so don't preload them\n",
    "ds_opts['aug_type'] = 'heavy'                         # Apply all supported augmentations\n",
    "ds_opts['batch_size'] = batch_size * len(gpu_devices) # Use a multiple of 8; here, 16 for dual-GPU mode (Titan X & 1080 Ti)\n",
    "ds_opts['crop_preproc'] = (256, 448)                  # Crop to a smaller input size\n",
    "ds1 = FlyingChairsDataset(mode='train_with_val', ds_root=_FLYINGCHAIRS_ROOT, options=ds_opts)\n",
    "ds_opts['type'] = 'into_future'\n",
    "ds2 = FlyingThings3DHalfResDataset(mode='train_with_val', ds_root=_FLYINGTHINGS3DHALFRES_ROOT, options=ds_opts)\n",
    "ds = MixedDataset(mode='train_with_val', datasets=[ds1, ds2], options=ds_opts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Dataset Configuration:\n",
      "  verbose              False\n",
      "  in_memory            False\n",
      "  crop_preproc         (256, 448)\n",
      "  scale_preproc        None\n",
      "  input_channels       3\n",
      "  tb_test_imgs         False\n",
      "  random_seed          1969\n",
      "  val_split            0.03\n",
      "  aug_type             heavy\n",
      "  aug_labels           True\n",
      "  fliplr               0.5\n",
      "  flipud               0.5\n",
      "  translate            (0.5, 0.05)\n",
      "  scale                (0.5, 0.05)\n",
      "  batch_size           16\n",
      "  type                 into_future\n",
      "  mode                 train_with_val\n",
      "  train size           41282\n",
      "  val size             1230\n"
     ]
    }
   ],
   "source": [
    "# Display dataset configuration\n",
    "ds.print_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start from the default options\n",
    "nn_opts = deepcopy(_DEFAULT_PWCNET_TRAIN_OPTIONS)\n",
    "nn_opts['verbose'] = True\n",
    "nn_opts['ckpt_dir'] = './pwcnet-sm-6-2-cyclic-chairsthingsmix/'\n",
    "nn_opts['batch_size'] = ds_opts['batch_size']\n",
    "nn_opts['x_shape'] = [2, ds_opts['crop_preproc'][0], ds_opts['crop_preproc'][1], 3]\n",
    "nn_opts['y_shape'] = [ds_opts['crop_preproc'][0], ds_opts['crop_preproc'][1], 2]\n",
    "nn_opts['use_tf_data'] = True # Use tf.data reader\n",
    "nn_opts['gpu_devices'] = gpu_devices\n",
    "nn_opts['controller'] = controller\n",
    "\n",
    "# Use the PWC-Net-small model in quarter-resolution mode\n",
    "nn_opts['use_dense_cx'] = False\n",
    "nn_opts['use_res_cx'] = False\n",
    "nn_opts['pyr_lvls'] = 6\n",
    "nn_opts['flow_pred_lvl'] = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the learning rate schedule. This schedule is for a single GPU using a batch size of 8.\n",
    "# Below,we adjust the schedule to the size of the batch and the number of GPUs.\n",
    "nn_opts['lr_policy'] = 'cyclic'\n",
    "nn_opts['cyclic_lr_max'] = 5e-04 # Anything higher will generate NaNs\n",
    "nn_opts['cyclic_lr_base'] = 1e-05\n",
    "nn_opts['cyclic_lr_stepsize'] = 20000\n",
    "nn_opts['max_steps'] = 200000\n",
    "\n",
    "# Below,we adjust the schedule to the size of the batch and our number of GPUs (2).\n",
    "nn_opts['max_steps'] = int(nn_opts['max_steps'] * 8 / ds_opts['batch_size'])\n",
    "nn_opts['cyclic_lr_stepsize'] = int(nn_opts['cyclic_lr_stepsize'] * 8 / ds_opts['batch_size'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building model towers...\n",
      "  Building tower_0...\n",
      "WARNING:tensorflow:From e:\\toolkits.win\\anaconda3-5.2.0\\envs\\dlwin36\\lib\\site-packages\\tensorflow\\python\\keras\\initializers.py:104: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with distribution=normal is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "`normal` is a deprecated alias for `truncated_normal`\n",
      "  ...tower_0 built.\n",
      "  Building tower_1...\n",
      "  ...tower_1 built.\n",
      "... model towers built.\n",
      "Initializing model with random values for initial training...\n",
      "\n",
      "... model initialized\n",
      "\n",
      "Model Configuration:\n",
      "  verbose                True\n",
      "  ckpt_dir               ./pwcnet-sm-6-2-cyclic-chairsthingsmix/\n",
      "  max_to_keep            10\n",
      "  x_dtype                <dtype: 'float32'>\n",
      "  x_shape                [2, 256, 448, 3]\n",
      "  y_dtype                <dtype: 'float32'>\n",
      "  y_shape                [256, 448, 2]\n",
      "  train_mode             train\n",
      "  display_step           100\n",
      "  snapshot_step          1000\n",
      "  val_step               1000\n",
      "  val_batch_size         -1\n",
      "  tb_val_imgs            pyramid\n",
      "  tb_test_imgs           None\n",
      "  gpu_devices            ['/device:GPU:0', '/device:GPU:1']\n",
      "  controller             /device:CPU:0\n",
      "  use_tf_data            True\n",
      "  batch_size             16\n",
      "  lr_policy              cyclic\n",
      "  max_steps              50000\n",
      "  cyclic_lr_max          0.0005\n",
      "  cyclic_lr_base         1e-05\n",
      "  cyclic_lr_stepsize     5000\n",
      "  loss_fn                loss_multiscale\n",
      "  alphas                 [0.32, 0.08, 0.02, 0.01, 0.005, 0.0025]\n",
      "  gamma                  0.0004\n",
      "  q                      1.0\n",
      "  epsilon                0.0\n",
      "  pyr_lvls               6\n",
      "  flow_pred_lvl          2\n",
      "  search_range           4\n",
      "  use_dense_cx           False\n",
      "  use_res_cx             False\n",
      "  mode                   train_with_val\n",
      "  trainable params       4705064\n"
     ]
    }
   ],
   "source": [
    "# Instantiate the model and display the model configuration\n",
    "nn = ModelPWCNet(mode='train_with_val', options=nn_opts, dataset=ds)\n",
    "nn.print_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training from scratch...\n",
      "2018-09-08 05:49:28 Iter 100 [Train]: loss=221.19, epe=18.03, lr=0.000020, samples/sec=23.2, sec/step=1.379, eta=19:06:58\n",
      "2018-09-08 05:50:39 Iter 200 [Train]: loss=217.13, epe=17.71, lr=0.000030, samples/sec=52.1, sec/step=0.614, eta=8:29:35\n",
      "2018-09-08 05:51:50 Iter 300 [Train]: loss=216.84, epe=17.69, lr=0.000039, samples/sec=52.3, sec/step=0.612, eta=8:27:17\n",
      "2018-09-08 05:53:01 Iter 400 [Train]: loss=220.07, epe=17.97, lr=0.000049, samples/sec=52.3, sec/step=0.612, eta=8:26:03\n",
      "2018-09-08 05:54:12 Iter 500 [Train]: loss=217.68, epe=17.78, lr=0.000059, samples/sec=52.3, sec/step=0.612, eta=8:25:12\n",
      "2018-09-08 05:55:23 Iter 600 [Train]: loss=221.48, epe=18.01, lr=0.000069, samples/sec=51.7, sec/step=0.619, eta=8:29:38\n",
      "2018-09-08 05:56:35 Iter 700 [Train]: loss=200.21, epe=15.82, lr=0.000079, samples/sec=51.7, sec/step=0.619, eta=8:28:21\n",
      "2018-09-08 05:57:48 Iter 800 [Train]: loss=193.49, epe=14.99, lr=0.000088, samples/sec=50.6, sec/step=0.632, eta=8:38:29\n",
      "2018-09-08 05:59:00 Iter 900 [Train]: loss=184.27, epe=14.24, lr=0.000098, samples/sec=50.6, sec/step=0.633, eta=8:37:46\n",
      "2018-09-08 06:00:13 Iter 1000 [Train]: loss=182.64, epe=14.08, lr=0.000108, samples/sec=50.3, sec/step=0.636, eta=8:39:03\n",
      "2018-09-08 06:00:33 Iter 1000 [Val]: loss=168.55, epe=13.22\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-1000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-1000\n",
      "2018-09-08 06:02:07 Iter 1100 [Train]: loss=181.22, epe=13.94, lr=0.000118, samples/sec=50.1, sec/step=0.639, eta=8:41:03\n",
      "2018-09-08 06:03:20 Iter 1200 [Train]: loss=180.56, epe=13.89, lr=0.000128, samples/sec=49.9, sec/step=0.641, eta=8:41:07\n",
      "2018-09-08 06:04:34 Iter 1300 [Train]: loss=176.08, epe=13.47, lr=0.000137, samples/sec=49.9, sec/step=0.641, eta=8:40:37\n",
      "2018-09-08 06:05:46 Iter 1400 [Train]: loss=171.07, epe=13.06, lr=0.000147, samples/sec=50.5, sec/step=0.634, eta=8:33:09\n",
      "2018-09-08 06:06:59 Iter 1500 [Train]: loss=170.79, epe=13.09, lr=0.000157, samples/sec=50.4, sec/step=0.635, eta=8:33:20\n",
      "2018-09-08 06:08:12 Iter 1600 [Train]: loss=171.24, epe=13.07, lr=0.000167, samples/sec=50.3, sec/step=0.636, eta=8:33:16\n",
      "2018-09-08 06:09:25 Iter 1700 [Train]: loss=162.46, epe=12.33, lr=0.000177, samples/sec=50.2, sec/step=0.638, eta=8:33:12\n",
      "2018-09-08 06:10:38 Iter 1800 [Train]: loss=162.10, epe=12.36, lr=0.000186, samples/sec=50.5, sec/step=0.634, eta=8:29:24\n",
      "2018-09-08 06:11:50 Iter 1900 [Train]: loss=160.26, epe=12.15, lr=0.000196, samples/sec=50.3, sec/step=0.636, eta=8:29:46\n",
      "2018-09-08 06:13:03 Iter 2000 [Train]: loss=151.30, epe=11.44, lr=0.000206, samples/sec=50.3, sec/step=0.636, eta=8:28:48\n",
      "2018-09-08 06:13:16 Iter 2000 [Val]: loss=150.77, epe=11.65\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-2000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-2000\n",
      "2018-09-08 06:14:44 Iter 2100 [Train]: loss=159.88, epe=12.22, lr=0.000216, samples/sec=50.5, sec/step=0.633, eta=8:25:40\n",
      "2018-09-08 06:15:57 Iter 2200 [Train]: loss=155.20, epe=11.82, lr=0.000226, samples/sec=50.0, sec/step=0.640, eta=8:29:58\n",
      "2018-09-08 06:17:10 Iter 2300 [Train]: loss=151.16, epe=11.51, lr=0.000235, samples/sec=49.7, sec/step=0.644, eta=8:32:01\n",
      "2018-09-08 06:18:24 Iter 2400 [Train]: loss=148.70, epe=11.34, lr=0.000245, samples/sec=49.8, sec/step=0.643, eta=8:29:43\n",
      "2018-09-08 06:19:37 Iter 2500 [Train]: loss=145.73, epe=11.11, lr=0.000255, samples/sec=49.9, sec/step=0.642, eta=8:28:10\n",
      "2018-09-08 06:20:50 Iter 2600 [Train]: loss=141.08, epe=10.76, lr=0.000265, samples/sec=49.9, sec/step=0.641, eta=8:26:15\n",
      "2018-09-08 06:22:03 Iter 2700 [Train]: loss=136.83, epe=10.40, lr=0.000275, samples/sec=50.3, sec/step=0.636, eta=8:21:12\n",
      "2018-09-08 06:23:16 Iter 2800 [Train]: loss=140.63, epe=10.72, lr=0.000284, samples/sec=50.2, sec/step=0.637, eta=8:21:13\n",
      "2018-09-08 06:24:29 Iter 2900 [Train]: loss=141.72, epe=10.86, lr=0.000294, samples/sec=50.1, sec/step=0.638, eta=8:21:10\n",
      "2018-09-08 06:25:41 Iter 3000 [Train]: loss=138.07, epe=10.56, lr=0.000304, samples/sec=50.3, sec/step=0.636, eta=8:18:32\n",
      "2018-09-08 06:25:54 Iter 3000 [Val]: loss=132.03, epe=10.24\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-3000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-3000\n",
      "2018-09-08 06:27:23 Iter 3100 [Train]: loss=144.42, epe=11.12, lr=0.000314, samples/sec=50.3, sec/step=0.636, eta=8:17:01\n",
      "2018-09-08 06:28:36 Iter 3200 [Train]: loss=128.88, epe=9.80, lr=0.000324, samples/sec=49.6, sec/step=0.645, eta=8:22:52\n",
      "2018-09-08 06:29:50 Iter 3300 [Train]: loss=133.21, epe=10.17, lr=0.000333, samples/sec=49.8, sec/step=0.643, eta=8:20:26\n",
      "2018-09-08 06:31:04 Iter 3400 [Train]: loss=133.16, epe=10.22, lr=0.000343, samples/sec=49.6, sec/step=0.645, eta=8:21:04\n",
      "2018-09-08 06:32:17 Iter 3500 [Train]: loss=128.64, epe=9.85, lr=0.000353, samples/sec=49.6, sec/step=0.645, eta=8:19:57\n",
      "2018-09-08 06:33:31 Iter 3600 [Train]: loss=129.11, epe=9.88, lr=0.000363, samples/sec=49.7, sec/step=0.644, eta=8:18:03\n",
      "2018-09-08 06:34:44 Iter 3700 [Train]: loss=129.77, epe=9.96, lr=0.000373, samples/sec=49.6, sec/step=0.645, eta=8:17:26\n",
      "2018-09-08 06:35:58 Iter 3800 [Train]: loss=129.96, epe=9.99, lr=0.000382, samples/sec=49.6, sec/step=0.645, eta=8:16:56\n",
      "2018-09-08 06:37:12 Iter 3900 [Train]: loss=127.20, epe=9.74, lr=0.000392, samples/sec=49.7, sec/step=0.644, eta=8:14:52\n",
      "2018-09-08 06:38:25 Iter 4000 [Train]: loss=125.49, epe=9.61, lr=0.000402, samples/sec=49.7, sec/step=0.643, eta=8:13:09\n",
      "2018-09-08 06:38:37 Iter 4000 [Val]: loss=122.45, epe=9.45\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-4000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-4000\n",
      "2018-09-08 06:40:07 Iter 4100 [Train]: loss=124.64, epe=9.54, lr=0.000412, samples/sec=50.0, sec/step=0.640, eta=8:09:51\n",
      "2018-09-08 06:41:20 Iter 4200 [Train]: loss=120.27, epe=9.18, lr=0.000422, samples/sec=49.5, sec/step=0.647, eta=8:13:40\n",
      "2018-09-08 06:42:34 Iter 4300 [Train]: loss=120.36, epe=9.18, lr=0.000431, samples/sec=49.4, sec/step=0.648, eta=8:13:26\n",
      "2018-09-08 06:43:48 Iter 4400 [Train]: loss=125.26, epe=9.60, lr=0.000441, samples/sec=49.5, sec/step=0.646, eta=8:11:00\n",
      "2018-09-08 06:45:01 Iter 4500 [Train]: loss=115.09, epe=8.79, lr=0.000451, samples/sec=49.6, sec/step=0.646, eta=8:09:35\n",
      "2018-09-08 06:46:15 Iter 4600 [Train]: loss=122.62, epe=9.36, lr=0.000461, samples/sec=49.4, sec/step=0.648, eta=8:10:13\n",
      "2018-09-08 06:47:29 Iter 4700 [Train]: loss=118.63, epe=9.06, lr=0.000471, samples/sec=49.2, sec/step=0.650, eta=8:11:05\n",
      "2018-09-08 06:48:43 Iter 4800 [Train]: loss=116.71, epe=8.90, lr=0.000480, samples/sec=49.3, sec/step=0.649, eta=8:08:58\n",
      "2018-09-08 06:49:57 Iter 4900 [Train]: loss=118.48, epe=9.04, lr=0.000490, samples/sec=49.3, sec/step=0.649, eta=8:07:49\n",
      "2018-09-08 06:51:11 Iter 5000 [Train]: loss=116.60, epe=8.93, lr=0.000500, samples/sec=49.5, sec/step=0.646, eta=8:04:52\n",
      "2018-09-08 06:51:23 Iter 5000 [Val]: loss=118.56, epe=9.25\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-5000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-5000\n",
      "2018-09-08 06:52:52 Iter 5100 [Train]: loss=115.03, epe=8.77, lr=0.000490, samples/sec=49.8, sec/step=0.642, eta=8:00:39\n",
      "2018-09-08 06:54:06 Iter 5200 [Train]: loss=114.75, epe=8.73, lr=0.000480, samples/sec=49.4, sec/step=0.648, eta=8:03:34\n",
      "2018-09-08 06:55:20 Iter 5300 [Train]: loss=111.01, epe=8.41, lr=0.000471, samples/sec=49.5, sec/step=0.647, eta=8:01:41\n",
      "2018-09-08 06:56:33 Iter 5400 [Train]: loss=106.40, epe=8.01, lr=0.000461, samples/sec=49.5, sec/step=0.647, eta=8:01:01\n",
      "2018-09-08 06:57:47 Iter 5500 [Train]: loss=107.73, epe=8.15, lr=0.000451, samples/sec=49.3, sec/step=0.650, eta=8:01:49\n",
      "2018-09-08 06:59:01 Iter 5600 [Train]: loss=106.29, epe=8.03, lr=0.000441, samples/sec=49.5, sec/step=0.647, eta=7:58:28\n",
      "2018-09-08 07:00:15 Iter 5700 [Train]: loss=110.01, epe=8.33, lr=0.000431, samples/sec=49.2, sec/step=0.651, eta=8:00:17\n",
      "2018-09-08 07:01:28 Iter 5800 [Train]: loss=111.52, epe=8.47, lr=0.000422, samples/sec=49.3, sec/step=0.649, eta=7:58:13\n",
      "2018-09-08 07:02:43 Iter 5900 [Train]: loss=105.96, epe=8.00, lr=0.000412, samples/sec=49.2, sec/step=0.650, eta=7:57:49\n",
      "2018-09-08 07:03:57 Iter 6000 [Train]: loss=103.04, epe=7.75, lr=0.000402, samples/sec=49.1, sec/step=0.652, eta=7:57:53\n",
      "2018-09-08 07:04:09 Iter 6000 [Val]: loss=103.63, epe=7.88\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-6000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-6000\n",
      "2018-09-08 07:05:40 Iter 6100 [Train]: loss=104.18, epe=7.83, lr=0.000392, samples/sec=49.2, sec/step=0.650, eta=7:55:51\n",
      "2018-09-08 07:06:55 Iter 6200 [Train]: loss=102.55, epe=7.67, lr=0.000382, samples/sec=48.6, sec/step=0.659, eta=8:00:46\n",
      "2018-09-08 07:08:09 Iter 6300 [Train]: loss=105.51, epe=7.94, lr=0.000373, samples/sec=48.8, sec/step=0.656, eta=7:57:57\n",
      "2018-09-08 07:09:24 Iter 6400 [Train]: loss=101.53, epe=7.59, lr=0.000363, samples/sec=48.7, sec/step=0.657, eta=7:57:32\n",
      "2018-09-08 07:10:39 Iter 6500 [Train]: loss=103.65, epe=7.77, lr=0.000353, samples/sec=48.6, sec/step=0.659, eta=7:57:34\n",
      "2018-09-08 07:11:54 Iter 6600 [Train]: loss=103.80, epe=7.77, lr=0.000343, samples/sec=48.8, sec/step=0.655, eta=7:54:07\n",
      "2018-09-08 07:13:08 Iter 6700 [Train]: loss=97.91, epe=7.31, lr=0.000333, samples/sec=48.8, sec/step=0.656, eta=7:53:40\n",
      "2018-09-08 07:14:23 Iter 6800 [Train]: loss=99.25, epe=7.39, lr=0.000324, samples/sec=48.8, sec/step=0.656, eta=7:52:36\n",
      "2018-09-08 07:15:37 Iter 6900 [Train]: loss=103.56, epe=7.78, lr=0.000314, samples/sec=48.7, sec/step=0.657, eta=7:51:59\n",
      "2018-09-08 07:16:52 Iter 7000 [Train]: loss=98.51, epe=7.34, lr=0.000304, samples/sec=48.7, sec/step=0.657, eta=7:50:59\n",
      "2018-09-08 07:17:05 Iter 7000 [Val]: loss=95.73, epe=7.20\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-7000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-7000\n",
      "2018-09-08 07:18:36 Iter 7100 [Train]: loss=96.78, epe=7.22, lr=0.000294, samples/sec=49.3, sec/step=0.649, eta=7:44:14\n",
      "2018-09-08 07:19:51 Iter 7200 [Train]: loss=97.88, epe=7.30, lr=0.000284, samples/sec=48.4, sec/step=0.662, eta=7:51:54\n",
      "2018-09-08 07:21:06 Iter 7300 [Train]: loss=97.48, epe=7.25, lr=0.000275, samples/sec=48.4, sec/step=0.661, eta=7:50:15\n",
      "2018-09-08 07:22:21 Iter 7400 [Train]: loss=94.01, epe=6.99, lr=0.000265, samples/sec=48.6, sec/step=0.659, eta=7:47:50\n",
      "2018-09-08 07:23:36 Iter 7500 [Train]: loss=95.39, epe=7.08, lr=0.000255, samples/sec=48.7, sec/step=0.658, eta=7:45:54\n",
      "2018-09-08 07:24:51 Iter 7600 [Train]: loss=94.09, epe=6.99, lr=0.000245, samples/sec=48.6, sec/step=0.659, eta=7:45:44\n",
      "2018-09-08 07:26:06 Iter 7700 [Train]: loss=99.92, epe=7.45, lr=0.000235, samples/sec=48.5, sec/step=0.660, eta=7:45:18\n",
      "2018-09-08 07:27:21 Iter 7800 [Train]: loss=90.34, epe=6.69, lr=0.000226, samples/sec=48.8, sec/step=0.655, eta=7:41:01\n",
      "2018-09-08 07:28:35 Iter 7900 [Train]: loss=91.41, epe=6.76, lr=0.000216, samples/sec=48.4, sec/step=0.661, eta=7:43:40\n",
      "2018-09-08 07:29:50 Iter 8000 [Train]: loss=92.07, epe=6.82, lr=0.000206, samples/sec=48.6, sec/step=0.658, eta=7:40:36\n",
      "2018-09-08 07:30:03 Iter 8000 [Val]: loss=93.31, epe=6.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-8000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-8000\n",
      "2018-09-08 07:31:34 Iter 8100 [Train]: loss=90.95, epe=6.73, lr=0.000196, samples/sec=48.7, sec/step=0.657, eta=7:38:28\n",
      "2018-09-08 07:32:49 Iter 8200 [Train]: loss=91.86, epe=6.79, lr=0.000186, samples/sec=48.3, sec/step=0.662, eta=7:41:12\n",
      "2018-09-08 07:34:04 Iter 8300 [Train]: loss=91.56, epe=6.79, lr=0.000177, samples/sec=48.3, sec/step=0.663, eta=7:40:27\n",
      "2018-09-08 07:35:20 Iter 8400 [Train]: loss=91.26, epe=6.72, lr=0.000167, samples/sec=48.1, sec/step=0.665, eta=7:40:56\n",
      "2018-09-08 07:36:35 Iter 8500 [Train]: loss=89.55, epe=6.62, lr=0.000157, samples/sec=48.4, sec/step=0.662, eta=7:37:41\n",
      "2018-09-08 07:37:50 Iter 8600 [Train]: loss=90.95, epe=6.71, lr=0.000147, samples/sec=48.4, sec/step=0.662, eta=7:36:37\n",
      "2018-09-08 07:39:06 Iter 8700 [Train]: loss=87.78, epe=6.45, lr=0.000137, samples/sec=48.2, sec/step=0.664, eta=7:37:09\n",
      "2018-09-08 07:40:21 Iter 8800 [Train]: loss=84.28, epe=6.18, lr=0.000128, samples/sec=48.3, sec/step=0.662, eta=7:34:36\n",
      "2018-09-08 07:41:36 Iter 8900 [Train]: loss=91.44, epe=6.77, lr=0.000118, samples/sec=48.1, sec/step=0.665, eta=7:35:32\n",
      "2018-09-08 07:42:51 Iter 9000 [Train]: loss=88.67, epe=6.54, lr=0.000108, samples/sec=48.5, sec/step=0.660, eta=7:31:04\n",
      "2018-09-08 07:43:04 Iter 9000 [Val]: loss=88.35, epe=6.57\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-9000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-9000\n",
      "2018-09-08 07:44:35 Iter 9100 [Train]: loss=86.84, epe=6.39, lr=0.000098, samples/sec=48.4, sec/step=0.661, eta=7:30:41\n",
      "2018-09-08 07:45:51 Iter 9200 [Train]: loss=86.40, epe=6.36, lr=0.000088, samples/sec=48.0, sec/step=0.667, eta=7:33:41\n",
      "2018-09-08 07:47:07 Iter 9300 [Train]: loss=85.06, epe=6.23, lr=0.000079, samples/sec=47.9, sec/step=0.668, eta=7:33:20\n",
      "2018-09-08 07:48:22 Iter 9400 [Train]: loss=86.95, epe=6.38, lr=0.000069, samples/sec=48.0, sec/step=0.667, eta=7:31:03\n",
      "2018-09-08 07:49:38 Iter 9500 [Train]: loss=84.91, epe=6.21, lr=0.000059, samples/sec=48.1, sec/step=0.665, eta=7:29:04\n",
      "2018-09-08 07:50:53 Iter 9600 [Train]: loss=86.17, epe=6.33, lr=0.000049, samples/sec=48.0, sec/step=0.667, eta=7:29:06\n",
      "2018-09-08 07:52:09 Iter 9700 [Train]: loss=86.88, epe=6.39, lr=0.000039, samples/sec=48.0, sec/step=0.667, eta=7:28:06\n",
      "2018-09-08 07:53:25 Iter 9800 [Train]: loss=81.31, epe=5.92, lr=0.000030, samples/sec=47.9, sec/step=0.668, eta=7:27:22\n",
      "2018-09-08 07:54:41 Iter 9900 [Train]: loss=85.22, epe=6.25, lr=0.000020, samples/sec=48.1, sec/step=0.666, eta=7:24:59\n",
      "2018-09-08 07:55:56 Iter 10000 [Train]: loss=84.21, epe=6.15, lr=0.000010, samples/sec=48.1, sec/step=0.666, eta=7:23:45\n",
      "2018-09-08 07:56:09 Iter 10000 [Val]: loss=83.92, epe=6.21\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-10000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-10000\n",
      "2018-09-08 07:57:41 Iter 10100 [Train]: loss=82.97, epe=6.05, lr=0.000015, samples/sec=47.8, sec/step=0.669, eta=7:24:57\n",
      "2018-09-08 07:58:57 Iter 10200 [Train]: loss=85.67, epe=6.29, lr=0.000020, samples/sec=47.8, sec/step=0.669, eta=7:23:53\n",
      "2018-09-08 08:00:13 Iter 10300 [Train]: loss=82.00, epe=5.98, lr=0.000025, samples/sec=47.7, sec/step=0.671, eta=7:24:17\n",
      "2018-09-08 08:01:30 Iter 10400 [Train]: loss=83.79, epe=6.14, lr=0.000030, samples/sec=47.3, sec/step=0.676, eta=7:26:26\n",
      "2018-09-08 08:02:46 Iter 10500 [Train]: loss=85.65, epe=6.29, lr=0.000034, samples/sec=47.7, sec/step=0.671, eta=7:21:26\n",
      "2018-09-08 08:04:02 Iter 10600 [Train]: loss=82.74, epe=6.03, lr=0.000039, samples/sec=47.7, sec/step=0.671, eta=7:20:42\n",
      "2018-09-08 08:05:18 Iter 10700 [Train]: loss=82.27, epe=6.01, lr=0.000044, samples/sec=47.7, sec/step=0.671, eta=7:19:20\n",
      "2018-09-08 08:06:34 Iter 10800 [Train]: loss=82.08, epe=6.00, lr=0.000049, samples/sec=47.5, sec/step=0.674, eta=7:20:20\n",
      "2018-09-08 08:07:51 Iter 10900 [Train]: loss=82.36, epe=6.03, lr=0.000054, samples/sec=47.8, sec/step=0.670, eta=7:16:26\n",
      "2018-09-08 08:09:07 Iter 11000 [Train]: loss=86.28, epe=6.33, lr=0.000059, samples/sec=47.7, sec/step=0.671, eta=7:15:55\n",
      "2018-09-08 08:09:20 Iter 11000 [Val]: loss=86.97, epe=6.44\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-11000 is not in all_model_checkpoint_paths. Manually adding it.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-11000\n",
      "2018-09-08 08:10:53 Iter 11100 [Train]: loss=86.01, epe=6.29, lr=0.000064, samples/sec=47.8, sec/step=0.669, eta=7:13:39\n",
      "2018-09-08 08:12:10 Iter 11200 [Train]: loss=85.41, epe=6.26, lr=0.000069, samples/sec=47.3, sec/step=0.677, eta=7:17:46\n",
      "2018-09-08 08:13:27 Iter 11300 [Train]: loss=81.92, epe=5.98, lr=0.000074, samples/sec=47.3, sec/step=0.677, eta=7:16:43\n",
      "2018-09-08 08:14:44 Iter 11400 [Train]: loss=88.37, epe=6.49, lr=0.000079, samples/sec=47.4, sec/step=0.675, eta=7:14:20\n",
      "2018-09-08 08:16:01 Iter 11500 [Train]: loss=83.04, epe=6.07, lr=0.000083, samples/sec=47.4, sec/step=0.675, eta=7:13:19\n",
      "2018-09-08 08:17:18 Iter 11600 [Train]: loss=86.43, epe=6.34, lr=0.000088, samples/sec=47.2, sec/step=0.678, eta=7:14:12\n",
      "2018-09-08 08:18:35 Iter 11700 [Train]: loss=83.20, epe=6.10, lr=0.000093, samples/sec=47.0, sec/step=0.681, eta=7:14:37\n",
      "2018-09-08 08:19:51 Iter 11800 [Train]: loss=85.18, epe=6.23, lr=0.000098, samples/sec=47.3, sec/step=0.677, eta=7:11:10\n",
      "2018-09-08 08:21:08 Iter 11900 [Train]: loss=87.24, epe=6.41, lr=0.000103, samples/sec=47.2, sec/step=0.677, eta=7:10:03\n",
      "2018-09-08 08:22:25 Iter 12000 [Train]: loss=85.33, epe=6.26, lr=0.000108, samples/sec=47.1, sec/step=0.680, eta=7:10:29\n",
      "2018-09-08 08:22:38 Iter 12000 [Val]: loss=83.66, epe=6.18\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-12000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-12000\n",
      "2018-09-08 08:24:11 Iter 12100 [Train]: loss=81.83, epe=5.98, lr=0.000113, samples/sec=47.4, sec/step=0.675, eta=7:06:22\n",
      "2018-09-08 08:25:28 Iter 12200 [Train]: loss=84.26, epe=6.18, lr=0.000118, samples/sec=47.2, sec/step=0.679, eta=7:07:31\n",
      "2018-09-08 08:26:45 Iter 12300 [Train]: loss=86.91, epe=6.38, lr=0.000123, samples/sec=47.4, sec/step=0.675, eta=7:04:08\n",
      "2018-09-08 08:28:02 Iter 12400 [Train]: loss=83.38, epe=6.09, lr=0.000128, samples/sec=47.1, sec/step=0.679, eta=7:05:36\n",
      "2018-09-08 08:29:19 Iter 12500 [Train]: loss=81.48, epe=5.94, lr=0.000133, samples/sec=47.2, sec/step=0.678, eta=7:03:49\n",
      "2018-09-08 08:30:36 Iter 12600 [Train]: loss=88.37, epe=6.51, lr=0.000137, samples/sec=46.9, sec/step=0.682, eta=7:05:19\n",
      "2018-09-08 08:31:53 Iter 12700 [Train]: loss=82.63, epe=6.05, lr=0.000142, samples/sec=47.0, sec/step=0.680, eta=7:02:59\n",
      "2018-09-08 08:33:11 Iter 12800 [Train]: loss=85.07, epe=6.23, lr=0.000147, samples/sec=47.1, sec/step=0.680, eta=7:01:37\n",
      "2018-09-08 08:34:28 Iter 12900 [Train]: loss=86.47, epe=6.34, lr=0.000152, samples/sec=47.0, sec/step=0.680, eta=7:00:36\n",
      "2018-09-08 08:35:46 Iter 13000 [Train]: loss=85.26, epe=6.25, lr=0.000157, samples/sec=46.6, sec/step=0.686, eta=7:03:19\n",
      "2018-09-08 08:35:59 Iter 13000 [Val]: loss=86.09, epe=6.37\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-13000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-13000\n",
      "2018-09-08 08:37:33 Iter 13100 [Train]: loss=84.99, epe=6.20, lr=0.000162, samples/sec=46.7, sec/step=0.686, eta=7:01:43\n",
      "2018-09-08 08:38:51 Iter 13200 [Train]: loss=87.38, epe=6.40, lr=0.000167, samples/sec=46.6, sec/step=0.687, eta=7:01:12\n",
      "2018-09-08 08:40:09 Iter 13300 [Train]: loss=85.26, epe=6.23, lr=0.000172, samples/sec=46.7, sec/step=0.686, eta=6:59:31\n",
      "2018-09-08 08:41:26 Iter 13400 [Train]: loss=82.87, epe=6.05, lr=0.000177, samples/sec=46.5, sec/step=0.688, eta=6:59:40\n",
      "2018-09-08 08:42:44 Iter 13500 [Train]: loss=83.16, epe=6.07, lr=0.000182, samples/sec=46.7, sec/step=0.685, eta=6:56:53\n",
      "2018-09-08 08:44:02 Iter 13600 [Train]: loss=83.54, epe=6.09, lr=0.000186, samples/sec=46.8, sec/step=0.684, eta=6:55:00\n",
      "2018-09-08 08:45:19 Iter 13700 [Train]: loss=86.18, epe=6.29, lr=0.000191, samples/sec=47.0, sec/step=0.681, eta=6:51:59\n",
      "2018-09-08 08:46:37 Iter 13800 [Train]: loss=85.31, epe=6.25, lr=0.000196, samples/sec=46.8, sec/step=0.684, eta=6:52:51\n",
      "2018-09-08 08:47:54 Iter 13900 [Train]: loss=85.02, epe=6.19, lr=0.000201, samples/sec=46.7, sec/step=0.685, eta=6:52:05\n",
      "2018-09-08 08:49:12 Iter 14000 [Train]: loss=81.57, epe=5.93, lr=0.000206, samples/sec=46.9, sec/step=0.682, eta=6:49:10\n",
      "2018-09-08 08:49:25 Iter 14000 [Val]: loss=81.91, epe=5.99\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-14000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-14000\n",
      "2018-09-08 08:50:59 Iter 14100 [Train]: loss=88.73, epe=6.53, lr=0.000211, samples/sec=46.7, sec/step=0.686, eta=6:50:17\n",
      "2018-09-08 08:52:17 Iter 14200 [Train]: loss=83.64, epe=6.12, lr=0.000216, samples/sec=46.5, sec/step=0.689, eta=6:50:56\n",
      "2018-09-08 08:53:36 Iter 14300 [Train]: loss=86.23, epe=6.29, lr=0.000221, samples/sec=46.1, sec/step=0.694, eta=6:52:51\n",
      "2018-09-08 08:54:54 Iter 14400 [Train]: loss=83.58, epe=6.11, lr=0.000226, samples/sec=46.4, sec/step=0.689, eta=6:49:03\n",
      "2018-09-08 08:56:12 Iter 14500 [Train]: loss=78.96, epe=5.73, lr=0.000230, samples/sec=46.3, sec/step=0.690, eta=6:48:31\n",
      "2018-09-08 08:57:30 Iter 14600 [Train]: loss=86.09, epe=6.30, lr=0.000235, samples/sec=46.2, sec/step=0.693, eta=6:48:49\n",
      "2018-09-08 08:58:48 Iter 14700 [Train]: loss=84.79, epe=6.19, lr=0.000240, samples/sec=46.5, sec/step=0.687, eta=6:44:28\n",
      "2018-09-08 09:00:06 Iter 14800 [Train]: loss=88.55, epe=6.50, lr=0.000245, samples/sec=46.6, sec/step=0.687, eta=6:42:56\n",
      "2018-09-08 09:01:25 Iter 14900 [Train]: loss=86.72, epe=6.35, lr=0.000250, samples/sec=46.3, sec/step=0.692, eta=6:44:44\n",
      "2018-09-08 09:02:43 Iter 15000 [Train]: loss=83.76, epe=6.10, lr=0.000255, samples/sec=46.7, sec/step=0.685, eta=6:39:21\n",
      "2018-09-08 09:02:56 Iter 15000 [Val]: loss=89.58, epe=6.65\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-15000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-15000\n",
      "2018-09-08 09:04:32 Iter 15100 [Train]: loss=86.23, epe=6.31, lr=0.000250, samples/sec=46.8, sec/step=0.684, eta=6:38:08\n",
      "2018-09-08 09:05:50 Iter 15200 [Train]: loss=83.24, epe=6.07, lr=0.000245, samples/sec=46.3, sec/step=0.692, eta=6:41:13\n",
      "2018-09-08 09:07:09 Iter 15300 [Train]: loss=79.22, epe=5.74, lr=0.000240, samples/sec=46.1, sec/step=0.694, eta=6:41:23\n",
      "2018-09-08 09:08:27 Iter 15400 [Train]: loss=84.92, epe=6.22, lr=0.000235, samples/sec=46.2, sec/step=0.693, eta=6:39:33\n",
      "2018-09-08 09:09:46 Iter 15500 [Train]: loss=86.93, epe=6.33, lr=0.000231, samples/sec=46.2, sec/step=0.693, eta=6:38:31\n",
      "2018-09-08 09:11:05 Iter 15600 [Train]: loss=81.09, epe=5.88, lr=0.000226, samples/sec=45.8, sec/step=0.699, eta=6:40:51\n",
      "2018-09-08 09:12:23 Iter 15700 [Train]: loss=84.67, epe=6.17, lr=0.000221, samples/sec=46.1, sec/step=0.694, eta=6:36:28\n",
      "2018-09-08 09:13:42 Iter 15800 [Train]: loss=82.49, epe=6.00, lr=0.000216, samples/sec=46.0, sec/step=0.696, eta=6:36:29\n",
      "2018-09-08 09:15:00 Iter 15900 [Train]: loss=83.72, epe=6.09, lr=0.000211, samples/sec=46.3, sec/step=0.691, eta=6:32:54\n",
      "2018-09-08 09:16:19 Iter 16000 [Train]: loss=81.27, epe=5.88, lr=0.000206, samples/sec=46.2, sec/step=0.692, eta=6:32:17\n",
      "2018-09-08 09:16:32 Iter 16000 [Val]: loss=81.45, epe=5.98\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-16000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-16000\n",
      "2018-09-08 09:18:07 Iter 16100 [Train]: loss=80.62, epe=5.85, lr=0.000201, samples/sec=46.3, sec/step=0.692, eta=6:30:43\n",
      "2018-09-08 09:19:26 Iter 16200 [Train]: loss=81.83, epe=5.94, lr=0.000196, samples/sec=46.0, sec/step=0.696, eta=6:31:55\n",
      "2018-09-08 09:20:45 Iter 16300 [Train]: loss=79.61, epe=5.76, lr=0.000191, samples/sec=45.9, sec/step=0.697, eta=6:31:16\n",
      "2018-09-08 09:22:04 Iter 16400 [Train]: loss=82.08, epe=5.94, lr=0.000186, samples/sec=46.1, sec/step=0.693, eta=6:28:19\n",
      "2018-09-08 09:23:22 Iter 16500 [Train]: loss=80.96, epe=5.88, lr=0.000182, samples/sec=46.2, sec/step=0.693, eta=6:27:06\n",
      "2018-09-08 09:24:41 Iter 16600 [Train]: loss=80.31, epe=5.80, lr=0.000177, samples/sec=46.0, sec/step=0.695, eta=6:26:53\n",
      "2018-09-08 09:26:00 Iter 16700 [Train]: loss=82.13, epe=5.97, lr=0.000172, samples/sec=46.1, sec/step=0.694, eta=6:25:25\n",
      "2018-09-08 09:27:19 Iter 16800 [Train]: loss=76.56, epe=5.49, lr=0.000167, samples/sec=45.6, sec/step=0.701, eta=6:27:54\n",
      "2018-09-08 09:28:38 Iter 16900 [Train]: loss=76.22, epe=5.49, lr=0.000162, samples/sec=45.6, sec/step=0.702, eta=6:27:08\n",
      "2018-09-08 09:29:57 Iter 17000 [Train]: loss=74.74, epe=5.36, lr=0.000157, samples/sec=46.0, sec/step=0.695, eta=6:22:22\n",
      "2018-09-08 09:30:11 Iter 17000 [Val]: loss=80.62, epe=5.87\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-17000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-17000\n",
      "2018-09-08 09:31:45 Iter 17100 [Train]: loss=76.79, epe=5.52, lr=0.000152, samples/sec=46.2, sec/step=0.693, eta=6:20:11\n",
      "2018-09-08 09:33:05 Iter 17200 [Train]: loss=80.78, epe=5.84, lr=0.000147, samples/sec=45.6, sec/step=0.702, eta=6:23:45\n",
      "2018-09-08 09:34:24 Iter 17300 [Train]: loss=79.28, epe=5.73, lr=0.000142, samples/sec=45.8, sec/step=0.699, eta=6:20:55\n",
      "2018-09-08 09:35:44 Iter 17400 [Train]: loss=76.63, epe=5.51, lr=0.000137, samples/sec=45.4, sec/step=0.704, eta=6:22:35\n",
      "2018-09-08 09:37:03 Iter 17500 [Train]: loss=79.75, epe=5.77, lr=0.000133, samples/sec=46.0, sec/step=0.695, eta=6:16:33\n",
      "2018-09-08 09:38:22 Iter 17600 [Train]: loss=83.06, epe=6.02, lr=0.000128, samples/sec=45.9, sec/step=0.697, eta=6:16:38\n",
      "2018-09-08 09:39:41 Iter 17700 [Train]: loss=74.54, epe=nan, lr=0.000123, samples/sec=46.0, sec/step=0.696, eta=6:14:43\n",
      "2018-09-08 09:41:00 Iter 17800 [Train]: loss=76.95, epe=5.52, lr=0.000118, samples/sec=45.6, sec/step=0.701, eta=6:16:12\n",
      "2018-09-08 09:42:20 Iter 17900 [Train]: loss=78.72, epe=5.68, lr=0.000113, samples/sec=45.6, sec/step=0.702, eta=6:15:43\n",
      "2018-09-08 09:43:39 Iter 18000 [Train]: loss=77.20, epe=5.55, lr=0.000108, samples/sec=45.8, sec/step=0.699, eta=6:12:43\n",
      "2018-09-08 09:43:53 Iter 18000 [Val]: loss=80.20, epe=5.86\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-18000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-18000\n",
      "2018-09-08 09:45:28 Iter 18100 [Train]: loss=78.83, epe=5.69, lr=0.000103, samples/sec=45.7, sec/step=0.700, eta=6:11:59\n",
      "2018-09-08 09:46:48 Iter 18200 [Train]: loss=76.46, epe=5.49, lr=0.000098, samples/sec=45.1, sec/step=0.709, eta=6:15:49\n",
      "2018-09-08 09:48:09 Iter 18300 [Train]: loss=77.27, epe=5.55, lr=0.000093, samples/sec=44.9, sec/step=0.712, eta=6:16:23\n",
      "2018-09-08 09:49:29 Iter 18400 [Train]: loss=74.96, epe=5.37, lr=0.000088, samples/sec=45.2, sec/step=0.707, eta=6:12:36\n",
      "2018-09-08 09:50:50 Iter 18500 [Train]: loss=76.95, epe=5.52, lr=0.000084, samples/sec=44.9, sec/step=0.713, eta=6:14:20\n",
      "2018-09-08 09:52:10 Iter 18600 [Train]: loss=74.19, epe=5.28, lr=0.000079, samples/sec=45.2, sec/step=0.708, eta=6:10:35\n",
      "2018-09-08 09:53:29 Iter 18700 [Train]: loss=73.79, epe=5.27, lr=0.000074, samples/sec=45.4, sec/step=0.704, eta=6:07:30\n",
      "2018-09-08 09:54:49 Iter 18800 [Train]: loss=73.33, epe=5.24, lr=0.000069, samples/sec=45.2, sec/step=0.708, eta=6:08:14\n",
      "2018-09-08 09:56:10 Iter 18900 [Train]: loss=75.86, epe=5.41, lr=0.000064, samples/sec=45.3, sec/step=0.707, eta=6:06:30\n",
      "2018-09-08 09:57:30 Iter 19000 [Train]: loss=74.03, epe=5.28, lr=0.000059, samples/sec=45.1, sec/step=0.710, eta=6:06:46\n",
      "2018-09-08 09:57:44 Iter 19000 [Val]: loss=75.33, epe=5.42\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-19000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-19000\n",
      "2018-09-08 09:59:20 Iter 19100 [Train]: loss=73.15, epe=5.22, lr=0.000054, samples/sec=45.1, sec/step=0.709, eta=6:05:04\n",
      "2018-09-08 10:00:42 Iter 19200 [Train]: loss=73.69, epe=5.25, lr=0.000049, samples/sec=44.7, sec/step=0.716, eta=6:07:30\n",
      "2018-09-08 10:02:02 Iter 19300 [Train]: loss=75.79, epe=5.42, lr=0.000044, samples/sec=45.0, sec/step=0.711, eta=6:03:52\n",
      "2018-09-08 10:03:23 Iter 19400 [Train]: loss=73.04, epe=5.22, lr=0.000039, samples/sec=44.7, sec/step=0.717, eta=6:05:26\n",
      "2018-09-08 10:04:44 Iter 19500 [Train]: loss=71.53, epe=5.08, lr=0.000035, samples/sec=44.7, sec/step=0.716, eta=6:04:09\n",
      "2018-09-08 10:06:05 Iter 19600 [Train]: loss=75.75, epe=5.42, lr=0.000030, samples/sec=44.8, sec/step=0.714, eta=6:01:51\n",
      "2018-09-08 10:07:26 Iter 19700 [Train]: loss=74.65, epe=5.33, lr=0.000025, samples/sec=44.6, sec/step=0.717, eta=6:02:05\n",
      "2018-09-08 10:08:47 Iter 19800 [Train]: loss=71.41, epe=5.07, lr=0.000020, samples/sec=44.7, sec/step=0.716, eta=6:00:22\n",
      "2018-09-08 10:10:08 Iter 19900 [Train]: loss=71.04, epe=5.03, lr=0.000015, samples/sec=44.6, sec/step=0.717, eta=5:59:51\n",
      "2018-09-08 10:11:29 Iter 20000 [Train]: loss=71.77, epe=5.08, lr=0.000010, samples/sec=44.7, sec/step=0.717, eta=5:58:18\n",
      "2018-09-08 10:11:43 Iter 20000 [Val]: loss=74.60, epe=5.37\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-20000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-20000\n",
      "2018-09-08 10:13:19 Iter 20100 [Train]: loss=72.09, epe=5.11, lr=0.000012, samples/sec=45.0, sec/step=0.712, eta=5:54:40\n",
      "2018-09-08 10:14:40 Iter 20200 [Train]: loss=74.02, epe=5.28, lr=0.000015, samples/sec=44.5, sec/step=0.719, eta=5:57:00\n",
      "2018-09-08 10:16:02 Iter 20300 [Train]: loss=71.18, epe=5.07, lr=0.000017, samples/sec=44.4, sec/step=0.721, eta=5:57:07\n",
      "2018-09-08 10:17:23 Iter 20400 [Train]: loss=72.38, epe=5.14, lr=0.000020, samples/sec=44.6, sec/step=0.718, eta=5:54:08\n",
      "2018-09-08 10:18:45 Iter 20500 [Train]: loss=70.93, epe=5.04, lr=0.000022, samples/sec=44.4, sec/step=0.720, eta=5:54:09\n",
      "2018-09-08 10:20:06 Iter 20600 [Train]: loss=74.62, epe=5.31, lr=0.000025, samples/sec=44.6, sec/step=0.718, eta=5:51:42\n",
      "2018-09-08 10:21:27 Iter 20700 [Train]: loss=73.81, epe=5.27, lr=0.000027, samples/sec=44.5, sec/step=0.718, eta=5:50:50\n",
      "2018-09-08 10:22:48 Iter 20800 [Train]: loss=74.25, epe=5.29, lr=0.000030, samples/sec=44.3, sec/step=0.722, eta=5:51:34\n",
      "2018-09-08 10:24:10 Iter 20900 [Train]: loss=73.53, epe=5.25, lr=0.000032, samples/sec=44.4, sec/step=0.721, eta=5:49:27\n",
      "2018-09-08 10:25:31 Iter 21000 [Train]: loss=71.01, epe=5.04, lr=0.000034, samples/sec=44.4, sec/step=0.721, eta=5:48:37\n",
      "2018-09-08 10:25:46 Iter 21000 [Val]: loss=74.89, epe=5.38\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-21000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-21000\n",
      "2018-09-08 10:27:25 Iter 21100 [Train]: loss=72.85, epe=5.18, lr=0.000037, samples/sec=44.4, sec/step=0.722, eta=5:47:32\n",
      "2018-09-08 10:28:47 Iter 21200 [Train]: loss=72.63, epe=5.18, lr=0.000039, samples/sec=43.9, sec/step=0.729, eta=5:49:41\n",
      "2018-09-08 10:30:09 Iter 21300 [Train]: loss=71.88, epe=5.11, lr=0.000042, samples/sec=44.0, sec/step=0.727, eta=5:47:37\n",
      "2018-09-08 10:31:31 Iter 21400 [Train]: loss=73.32, epe=5.22, lr=0.000044, samples/sec=44.3, sec/step=0.723, eta=5:44:41\n",
      "2018-09-08 10:32:53 Iter 21500 [Train]: loss=74.34, epe=5.32, lr=0.000047, samples/sec=44.0, sec/step=0.728, eta=5:45:47\n",
      "2018-09-08 10:34:16 Iter 21600 [Train]: loss=73.79, epe=5.26, lr=0.000049, samples/sec=44.1, sec/step=0.725, eta=5:43:24\n",
      "2018-09-08 10:35:38 Iter 21700 [Train]: loss=70.31, epe=4.98, lr=0.000052, samples/sec=43.9, sec/step=0.728, eta=5:43:30\n",
      "2018-09-08 10:36:59 Iter 21800 [Train]: loss=70.68, epe=5.03, lr=0.000054, samples/sec=44.5, sec/step=0.719, eta=5:38:04\n",
      "2018-09-08 10:38:21 Iter 21900 [Train]: loss=73.14, epe=5.20, lr=0.000057, samples/sec=44.0, sec/step=0.727, eta=5:40:30\n",
      "2018-09-08 10:39:44 Iter 22000 [Train]: loss=73.60, epe=5.24, lr=0.000059, samples/sec=43.7, sec/step=0.731, eta=5:41:20\n",
      "2018-09-08 10:39:58 Iter 22000 [Val]: loss=72.40, epe=5.20\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-22000 is not in all_model_checkpoint_paths. Manually adding it.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-22000\n",
      "2018-09-08 10:41:36 Iter 22100 [Train]: loss=74.32, epe=5.30, lr=0.000061, samples/sec=44.0, sec/step=0.727, eta=5:38:10\n",
      "2018-09-08 10:42:59 Iter 22200 [Train]: loss=72.02, epe=5.12, lr=0.000064, samples/sec=43.5, sec/step=0.735, eta=5:40:30\n",
      "2018-09-08 10:44:22 Iter 22300 [Train]: loss=73.88, epe=5.29, lr=0.000066, samples/sec=43.5, sec/step=0.735, eta=5:39:33\n",
      "2018-09-08 10:45:44 Iter 22400 [Train]: loss=72.51, epe=5.15, lr=0.000069, samples/sec=43.5, sec/step=0.735, eta=5:38:05\n",
      "2018-09-08 10:47:07 Iter 22500 [Train]: loss=75.97, epe=5.44, lr=0.000071, samples/sec=43.6, sec/step=0.734, eta=5:36:30\n",
      "2018-09-08 10:48:30 Iter 22600 [Train]: loss=72.29, epe=5.14, lr=0.000074, samples/sec=43.4, sec/step=0.738, eta=5:36:50\n",
      "2018-09-08 10:49:53 Iter 22700 [Train]: loss=72.44, epe=5.13, lr=0.000076, samples/sec=43.7, sec/step=0.732, eta=5:32:59\n",
      "2018-09-08 10:51:16 Iter 22800 [Train]: loss=73.29, epe=5.23, lr=0.000079, samples/sec=43.8, sec/step=0.731, eta=5:31:17\n",
      "2018-09-08 10:52:39 Iter 22900 [Train]: loss=73.51, epe=5.23, lr=0.000081, samples/sec=43.6, sec/step=0.734, eta=5:31:28\n",
      "2018-09-08 10:54:02 Iter 23000 [Train]: loss=76.06, epe=5.45, lr=0.000083, samples/sec=43.5, sec/step=0.736, eta=5:31:02\n",
      "2018-09-08 10:54:16 Iter 23000 [Val]: loss=76.29, epe=5.51\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-23000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-23000\n",
      "2018-09-08 10:55:54 Iter 23100 [Train]: loss=71.76, epe=5.11, lr=0.000086, samples/sec=43.9, sec/step=0.729, eta=5:26:57\n",
      "2018-09-08 10:57:18 Iter 23200 [Train]: loss=74.79, epe=5.35, lr=0.000088, samples/sec=43.3, sec/step=0.740, eta=5:30:21\n",
      "2018-09-08 10:58:41 Iter 23300 [Train]: loss=73.32, epe=5.23, lr=0.000091, samples/sec=43.2, sec/step=0.740, eta=5:29:27\n",
      "2018-09-08 11:00:04 Iter 23400 [Train]: loss=75.09, epe=5.36, lr=0.000093, samples/sec=43.4, sec/step=0.737, eta=5:26:57\n",
      "2018-09-08 11:01:28 Iter 23500 [Train]: loss=74.60, epe=5.33, lr=0.000096, samples/sec=43.2, sec/step=0.741, eta=5:27:08\n",
      "2018-09-08 11:02:52 Iter 23600 [Train]: loss=75.74, epe=5.43, lr=0.000098, samples/sec=43.0, sec/step=0.744, eta=5:27:14\n",
      "2018-09-08 11:04:15 Iter 23700 [Train]: loss=75.31, epe=5.37, lr=0.000101, samples/sec=43.6, sec/step=0.734, eta=5:21:47\n",
      "2018-09-08 11:05:38 Iter 23800 [Train]: loss=77.25, epe=5.53, lr=0.000103, samples/sec=43.2, sec/step=0.741, eta=5:23:34\n",
      "2018-09-08 11:07:01 Iter 23900 [Train]: loss=72.25, epe=5.14, lr=0.000106, samples/sec=43.7, sec/step=0.733, eta=5:18:38\n",
      "2018-09-08 11:08:24 Iter 24000 [Train]: loss=72.16, epe=5.14, lr=0.000108, samples/sec=43.6, sec/step=0.733, eta=5:17:45\n",
      "2018-09-08 11:08:38 Iter 24000 [Val]: loss=75.81, epe=5.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-24000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-24000\n",
      "2018-09-08 11:10:17 Iter 24100 [Train]: loss=70.85, epe=5.04, lr=0.000110, samples/sec=43.4, sec/step=0.737, eta=5:18:09\n",
      "2018-09-08 11:11:41 Iter 24200 [Train]: loss=71.89, epe=5.11, lr=0.000113, samples/sec=42.9, sec/step=0.746, eta=5:20:39\n",
      "2018-09-08 11:13:05 Iter 24300 [Train]: loss=76.27, epe=5.47, lr=0.000115, samples/sec=43.1, sec/step=0.743, eta=5:18:18\n",
      "2018-09-08 11:14:29 Iter 24400 [Train]: loss=72.87, epe=5.19, lr=0.000118, samples/sec=43.0, sec/step=0.744, eta=5:17:24\n",
      "2018-09-08 11:15:53 Iter 24500 [Train]: loss=73.98, epe=5.29, lr=0.000120, samples/sec=43.1, sec/step=0.742, eta=5:15:15\n",
      "2018-09-08 11:17:16 Iter 24600 [Train]: loss=75.86, epe=5.42, lr=0.000123, samples/sec=43.0, sec/step=0.745, eta=5:15:17\n",
      "2018-09-08 11:18:40 Iter 24700 [Train]: loss=73.68, epe=5.25, lr=0.000125, samples/sec=42.9, sec/step=0.747, eta=5:14:47\n",
      "2018-09-08 11:20:05 Iter 24800 [Train]: loss=72.13, epe=5.13, lr=0.000128, samples/sec=42.6, sec/step=0.751, eta=5:15:35\n",
      "2018-09-08 11:21:29 Iter 24900 [Train]: loss=74.61, epe=5.31, lr=0.000130, samples/sec=42.8, sec/step=0.748, eta=5:13:03\n",
      "2018-09-08 11:22:53 Iter 25000 [Train]: loss=72.08, epe=5.13, lr=0.000133, samples/sec=43.0, sec/step=0.744, eta=5:09:49\n",
      "2018-09-08 11:23:08 Iter 25000 [Val]: loss=78.55, epe=5.68\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-25000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-25000\n",
      "2018-09-08 11:24:47 Iter 25100 [Train]: loss=77.12, epe=5.52, lr=0.000130, samples/sec=43.0, sec/step=0.743, eta=5:08:30\n",
      "2018-09-08 11:26:11 Iter 25200 [Train]: loss=76.21, epe=5.45, lr=0.000128, samples/sec=42.8, sec/step=0.747, eta=5:08:46\n",
      "2018-09-08 11:27:36 Iter 25300 [Train]: loss=73.98, epe=5.27, lr=0.000125, samples/sec=42.9, sec/step=0.746, eta=5:06:56\n",
      "2018-09-08 11:29:00 Iter 25400 [Train]: loss=73.95, epe=5.26, lr=0.000123, samples/sec=42.6, sec/step=0.751, eta=5:07:43\n",
      "2018-09-08 11:30:25 Iter 25500 [Train]: loss=70.35, epe=4.99, lr=0.000120, samples/sec=42.8, sec/step=0.748, eta=5:05:33\n",
      "2018-09-08 11:31:49 Iter 25600 [Train]: loss=72.48, epe=5.15, lr=0.000118, samples/sec=42.9, sec/step=0.746, eta=5:03:19\n",
      "2018-09-08 11:33:14 Iter 25700 [Train]: loss=74.59, epe=5.33, lr=0.000115, samples/sec=42.4, sec/step=0.755, eta=5:05:37\n",
      "2018-09-08 11:34:38 Iter 25800 [Train]: loss=73.81, epe=5.26, lr=0.000113, samples/sec=42.7, sec/step=0.750, eta=5:02:28\n",
      "2018-09-08 11:36:03 Iter 25900 [Train]: loss=70.45, epe=4.98, lr=0.000110, samples/sec=42.6, sec/step=0.751, eta=5:01:48\n",
      "2018-09-08 11:37:27 Iter 26000 [Train]: loss=74.29, epe=5.31, lr=0.000108, samples/sec=42.6, sec/step=0.751, eta=5:00:29\n",
      "2018-09-08 11:37:42 Iter 26000 [Val]: loss=71.34, epe=5.14\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-26000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-26000\n",
      "2018-09-08 11:39:22 Iter 26100 [Train]: loss=74.17, epe=5.31, lr=0.000106, samples/sec=42.6, sec/step=0.751, eta=4:59:20\n",
      "2018-09-08 11:40:48 Iter 26200 [Train]: loss=73.00, epe=5.18, lr=0.000103, samples/sec=41.8, sec/step=0.765, eta=5:03:27\n",
      "2018-09-08 11:42:14 Iter 26300 [Train]: loss=73.00, epe=5.19, lr=0.000101, samples/sec=42.1, sec/step=0.760, eta=5:00:20\n",
      "2018-09-08 11:43:40 Iter 26400 [Train]: loss=71.52, epe=5.06, lr=0.000098, samples/sec=41.5, sec/step=0.771, eta=5:03:04\n",
      "2018-09-08 11:45:06 Iter 26500 [Train]: loss=73.54, epe=5.23, lr=0.000096, samples/sec=41.9, sec/step=0.763, eta=4:58:51\n",
      "2018-09-08 11:46:33 Iter 26600 [Train]: loss=77.32, epe=5.53, lr=0.000093, samples/sec=41.9, sec/step=0.763, eta=4:57:42\n",
      "2018-09-08 11:48:00 Iter 26700 [Train]: loss=71.24, epe=5.03, lr=0.000091, samples/sec=41.2, sec/step=0.777, eta=5:01:38\n",
      "2018-09-08 11:49:26 Iter 26800 [Train]: loss=67.71, epe=4.77, lr=0.000088, samples/sec=42.0, sec/step=0.762, eta=4:54:36\n",
      "2018-09-08 11:50:52 Iter 26900 [Train]: loss=70.20, epe=4.96, lr=0.000086, samples/sec=42.3, sec/step=0.757, eta=4:51:23\n",
      "2018-09-08 11:52:17 Iter 27000 [Train]: loss=71.58, epe=5.06, lr=0.000083, samples/sec=42.4, sec/step=0.755, eta=4:49:15\n",
      "2018-09-08 11:52:32 Iter 27000 [Val]: loss=75.06, epe=5.35\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-27000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-27000\n",
      "2018-09-08 11:54:18 Iter 27100 [Train]: loss=69.45, epe=4.90, lr=0.000081, samples/sec=42.9, sec/step=0.746, eta=4:44:48\n",
      "2018-09-08 11:55:43 Iter 27200 [Train]: loss=70.50, epe=4.98, lr=0.000079, samples/sec=42.2, sec/step=0.759, eta=4:48:19\n",
      "2018-09-08 11:57:09 Iter 27300 [Train]: loss=72.32, epe=5.12, lr=0.000076, samples/sec=42.2, sec/step=0.758, eta=4:46:50\n",
      "2018-09-08 11:58:34 Iter 27400 [Train]: loss=72.19, epe=5.11, lr=0.000074, samples/sec=42.5, sec/step=0.752, eta=4:43:23\n",
      "2018-09-08 11:59:59 Iter 27500 [Train]: loss=68.33, epe=4.83, lr=0.000071, samples/sec=42.3, sec/step=0.756, eta=4:43:39\n",
      "2018-09-08 12:01:24 Iter 27600 [Train]: loss=72.06, epe=5.11, lr=0.000069, samples/sec=42.5, sec/step=0.753, eta=4:41:16\n",
      "2018-09-08 12:02:49 Iter 27700 [Train]: loss=72.03, epe=5.09, lr=0.000066, samples/sec=42.5, sec/step=0.752, eta=4:39:40\n",
      "2018-09-08 12:04:15 Iter 27800 [Train]: loss=69.75, epe=4.91, lr=0.000064, samples/sec=42.1, sec/step=0.760, eta=4:41:14\n",
      "2018-09-08 12:05:40 Iter 27900 [Train]: loss=70.17, epe=4.95, lr=0.000061, samples/sec=42.5, sec/step=0.753, eta=4:37:14\n",
      "2018-09-08 12:07:06 Iter 28000 [Train]: loss=68.67, epe=4.83, lr=0.000059, samples/sec=42.3, sec/step=0.757, eta=4:37:37\n",
      "2018-09-08 12:07:21 Iter 28000 [Val]: loss=72.05, epe=5.15\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-28000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-28000\n",
      "2018-09-08 12:09:05 Iter 28100 [Train]: loss=71.22, epe=5.04, lr=0.000057, samples/sec=42.0, sec/step=0.763, eta=4:38:25\n",
      "2018-09-08 12:10:32 Iter 28200 [Train]: loss=70.27, epe=4.96, lr=0.000054, samples/sec=41.5, sec/step=0.770, eta=4:39:56\n",
      "2018-09-08 12:11:59 Iter 28300 [Train]: loss=68.27, epe=4.79, lr=0.000052, samples/sec=41.3, sec/step=0.774, eta=4:39:56\n",
      "2018-09-08 12:13:28 Iter 28400 [Train]: loss=69.66, epe=4.92, lr=0.000049, samples/sec=40.7, sec/step=0.787, eta=4:43:23\n",
      "2018-09-08 12:14:56 Iter 28500 [Train]: loss=68.64, epe=4.83, lr=0.000047, samples/sec=40.5, sec/step=0.790, eta=4:42:57\n",
      "2018-09-08 12:16:23 Iter 28600 [Train]: loss=67.87, epe=4.77, lr=0.000044, samples/sec=41.7, sec/step=0.767, eta=4:33:32\n",
      "2018-09-08 12:17:49 Iter 28700 [Train]: loss=72.80, epe=5.16, lr=0.000042, samples/sec=42.1, sec/step=0.759, eta=4:29:31\n",
      "2018-09-08 12:19:15 Iter 28800 [Train]: loss=67.56, epe=4.73, lr=0.000039, samples/sec=41.8, sec/step=0.766, eta=4:30:40\n",
      "2018-09-08 12:20:41 Iter 28900 [Train]: loss=67.65, epe=4.74, lr=0.000037, samples/sec=41.8, sec/step=0.765, eta=4:29:02\n",
      "2018-09-08 12:22:08 Iter 29000 [Train]: loss=68.90, epe=4.84, lr=0.000035, samples/sec=41.8, sec/step=0.765, eta=4:27:55\n",
      "2018-09-08 12:22:23 Iter 29000 [Val]: loss=66.68, epe=4.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-29000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-29000\n",
      "2018-09-08 12:24:07 Iter 29100 [Train]: loss=68.32, epe=4.79, lr=0.000032, samples/sec=41.8, sec/step=0.766, eta=4:26:49\n",
      "2018-09-08 12:25:34 Iter 29200 [Train]: loss=73.21, epe=5.18, lr=0.000030, samples/sec=41.7, sec/step=0.767, eta=4:25:46\n",
      "2018-09-08 12:27:00 Iter 29300 [Train]: loss=68.53, epe=4.84, lr=0.000027, samples/sec=41.8, sec/step=0.765, eta=4:23:57\n",
      "2018-09-08 12:28:27 Iter 29400 [Train]: loss=69.20, epe=4.87, lr=0.000025, samples/sec=41.7, sec/step=0.767, eta=4:23:25\n",
      "2018-09-08 12:29:53 Iter 29500 [Train]: loss=66.72, epe=4.65, lr=0.000022, samples/sec=41.9, sec/step=0.763, eta=4:20:44\n",
      "2018-09-08 12:31:20 Iter 29600 [Train]: loss=68.81, epe=4.84, lr=0.000020, samples/sec=41.6, sec/step=0.769, eta=4:21:23\n",
      "2018-09-08 12:32:46 Iter 29700 [Train]: loss=70.17, epe=4.93, lr=0.000017, samples/sec=41.8, sec/step=0.765, eta=4:18:47\n",
      "2018-09-08 12:34:13 Iter 29800 [Train]: loss=68.88, epe=4.83, lr=0.000015, samples/sec=41.9, sec/step=0.764, eta=4:17:10\n",
      "2018-09-08 12:35:40 Iter 29900 [Train]: loss=65.74, epe=4.59, lr=0.000012, samples/sec=41.5, sec/step=0.770, eta=4:18:07\n",
      "2018-09-08 12:37:07 Iter 30000 [Train]: loss=68.32, epe=4.78, lr=0.000010, samples/sec=41.5, sec/step=0.771, eta=4:17:02\n",
      "2018-09-08 12:37:22 Iter 30000 [Val]: loss=73.13, epe=5.19\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-30000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-30000\n",
      "2018-09-08 12:39:06 Iter 30100 [Train]: loss=66.32, epe=4.63, lr=0.000011, samples/sec=42.0, sec/step=0.762, eta=4:12:37\n",
      "2018-09-08 12:40:33 Iter 30200 [Train]: loss=69.15, epe=4.85, lr=0.000012, samples/sec=41.6, sec/step=0.769, eta=4:13:37\n",
      "2018-09-08 12:42:00 Iter 30300 [Train]: loss=69.19, epe=4.86, lr=0.000014, samples/sec=41.7, sec/step=0.767, eta=4:11:43\n",
      "2018-09-08 12:43:26 Iter 30400 [Train]: loss=68.47, epe=4.81, lr=0.000015, samples/sec=41.7, sec/step=0.768, eta=4:10:48\n",
      "2018-09-08 12:44:53 Iter 30500 [Train]: loss=66.93, epe=4.69, lr=0.000016, samples/sec=41.5, sec/step=0.770, eta=4:10:18\n",
      "2018-09-08 12:46:20 Iter 30600 [Train]: loss=69.04, epe=4.84, lr=0.000017, samples/sec=41.7, sec/step=0.768, eta=4:08:13\n",
      "2018-09-08 12:47:47 Iter 30700 [Train]: loss=67.93, epe=4.77, lr=0.000019, samples/sec=41.4, sec/step=0.773, eta=4:08:32\n",
      "2018-09-08 12:49:14 Iter 30800 [Train]: loss=69.88, epe=4.91, lr=0.000020, samples/sec=41.8, sec/step=0.766, eta=4:04:59\n",
      "2018-09-08 12:50:41 Iter 30900 [Train]: loss=67.32, epe=4.71, lr=0.000021, samples/sec=41.5, sec/step=0.772, eta=4:05:42\n",
      "2018-09-08 12:52:08 Iter 31000 [Train]: loss=64.67, epe=4.52, lr=0.000022, samples/sec=41.5, sec/step=0.771, eta=4:04:03\n",
      "2018-09-08 12:52:24 Iter 31000 [Val]: loss=68.95, epe=4.88\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-31000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-31000\n",
      "2018-09-08 12:54:08 Iter 31100 [Train]: loss=69.36, epe=4.86, lr=0.000023, samples/sec=41.7, sec/step=0.767, eta=4:01:45\n",
      "2018-09-08 12:55:35 Iter 31200 [Train]: loss=68.84, epe=4.86, lr=0.000025, samples/sec=41.3, sec/step=0.776, eta=4:03:00\n",
      "2018-09-08 12:57:03 Iter 31300 [Train]: loss=66.72, epe=4.65, lr=0.000026, samples/sec=41.2, sec/step=0.776, eta=4:01:54\n",
      "2018-09-08 12:58:30 Iter 31400 [Train]: loss=70.76, epe=4.99, lr=0.000027, samples/sec=41.5, sec/step=0.771, eta=3:59:08\n",
      "2018-09-08 12:59:57 Iter 31500 [Train]: loss=67.20, epe=4.72, lr=0.000028, samples/sec=41.4, sec/step=0.773, eta=3:58:26\n",
      "2018-09-08 13:01:24 Iter 31600 [Train]: loss=68.08, epe=4.78, lr=0.000030, samples/sec=41.8, sec/step=0.765, eta=3:54:38\n",
      "2018-09-08 13:02:52 Iter 31700 [Train]: loss=68.58, epe=4.82, lr=0.000031, samples/sec=41.2, sec/step=0.778, eta=3:57:09\n",
      "2018-09-08 13:04:18 Iter 31800 [Train]: loss=67.72, epe=4.75, lr=0.000032, samples/sec=41.7, sec/step=0.767, eta=3:52:38\n",
      "2018-09-08 13:05:45 Iter 31900 [Train]: loss=69.51, epe=4.89, lr=0.000033, samples/sec=41.6, sec/step=0.770, eta=3:52:16\n",
      "2018-09-08 13:07:13 Iter 32000 [Train]: loss=67.22, epe=4.73, lr=0.000034, samples/sec=41.1, sec/step=0.779, eta=3:53:42\n",
      "2018-09-08 13:07:29 Iter 32000 [Val]: loss=69.01, epe=4.90\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-32000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-32000\n",
      "2018-09-08 13:09:14 Iter 32100 [Train]: loss=69.12, epe=4.85, lr=0.000036, samples/sec=41.3, sec/step=0.774, eta=3:51:03\n",
      "2018-09-08 13:10:42 Iter 32200 [Train]: loss=66.52, epe=4.64, lr=0.000037, samples/sec=41.0, sec/step=0.781, eta=3:51:33\n",
      "2018-09-08 13:12:10 Iter 32300 [Train]: loss=67.63, epe=4.76, lr=0.000038, samples/sec=40.8, sec/step=0.785, eta=3:51:35\n",
      "2018-09-08 13:13:38 Iter 32400 [Train]: loss=70.78, epe=5.00, lr=0.000039, samples/sec=41.2, sec/step=0.776, eta=3:47:42\n",
      "2018-09-08 13:15:06 Iter 32500 [Train]: loss=70.22, epe=4.93, lr=0.000041, samples/sec=40.8, sec/step=0.784, eta=3:48:42\n",
      "2018-09-08 13:16:34 Iter 32600 [Train]: loss=68.14, epe=4.80, lr=0.000042, samples/sec=41.1, sec/step=0.779, eta=3:45:52\n",
      "2018-09-08 13:18:02 Iter 32700 [Train]: loss=68.66, epe=4.80, lr=0.000043, samples/sec=40.9, sec/step=0.782, eta=3:45:23\n",
      "2018-09-08 13:19:30 Iter 32800 [Train]: loss=68.21, epe=4.78, lr=0.000044, samples/sec=41.2, sec/step=0.778, eta=3:42:54\n",
      "2018-09-08 13:20:58 Iter 32900 [Train]: loss=66.83, epe=4.65, lr=0.000046, samples/sec=41.3, sec/step=0.775, eta=3:40:51\n",
      "2018-09-08 13:22:25 Iter 33000 [Train]: loss=72.52, epe=5.14, lr=0.000047, samples/sec=41.1, sec/step=0.778, eta=3:40:23\n",
      "2018-09-08 13:22:41 Iter 33000 [Val]: loss=69.95, epe=4.96\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-33000 is not in all_model_checkpoint_paths. Manually adding it.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-33000\n",
      "2018-09-08 13:24:26 Iter 33100 [Train]: loss=67.33, epe=4.71, lr=0.000048, samples/sec=41.3, sec/step=0.774, eta=3:38:08\n",
      "2018-09-08 13:25:53 Iter 33200 [Train]: loss=67.43, epe=4.73, lr=0.000049, samples/sec=41.2, sec/step=0.776, eta=3:37:18\n",
      "2018-09-08 13:27:21 Iter 33300 [Train]: loss=66.81, epe=4.70, lr=0.000050, samples/sec=41.1, sec/step=0.779, eta=3:36:55\n",
      "2018-09-08 13:28:49 Iter 33400 [Train]: loss=68.18, epe=4.78, lr=0.000052, samples/sec=41.1, sec/step=0.778, eta=3:35:13\n",
      "2018-09-08 13:30:17 Iter 33500 [Train]: loss=68.39, epe=4.80, lr=0.000053, samples/sec=41.0, sec/step=0.781, eta=3:34:48\n",
      "2018-09-08 13:31:45 Iter 33600 [Train]: loss=66.06, epe=4.62, lr=0.000054, samples/sec=41.0, sec/step=0.781, eta=3:33:29\n",
      "2018-09-08 13:33:13 Iter 33700 [Train]: loss=68.08, epe=4.79, lr=0.000055, samples/sec=41.1, sec/step=0.779, eta=3:31:38\n",
      "2018-09-08 13:34:41 Iter 33800 [Train]: loss=65.89, epe=4.60, lr=0.000057, samples/sec=41.0, sec/step=0.781, eta=3:30:49\n",
      "2018-09-08 13:36:09 Iter 33900 [Train]: loss=68.13, epe=4.79, lr=0.000058, samples/sec=41.0, sec/step=0.780, eta=3:29:16\n",
      "2018-09-08 13:37:37 Iter 34000 [Train]: loss=68.12, epe=4.78, lr=0.000059, samples/sec=41.0, sec/step=0.781, eta=3:28:21\n",
      "2018-09-08 13:37:53 Iter 34000 [Val]: loss=73.01, epe=5.23\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-34000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-34000\n",
      "2018-09-08 13:39:38 Iter 34100 [Train]: loss=71.15, epe=5.04, lr=0.000060, samples/sec=40.9, sec/step=0.782, eta=3:27:17\n",
      "2018-09-08 13:41:07 Iter 34200 [Train]: loss=70.06, epe=4.93, lr=0.000061, samples/sec=40.8, sec/step=0.784, eta=3:26:34\n",
      "2018-09-08 13:42:35 Iter 34300 [Train]: loss=67.39, epe=4.71, lr=0.000063, samples/sec=40.9, sec/step=0.782, eta=3:24:38\n",
      "2018-09-08 13:44:04 Iter 34400 [Train]: loss=69.95, epe=4.92, lr=0.000064, samples/sec=40.7, sec/step=0.787, eta=3:24:33\n",
      "2018-09-08 13:45:33 Iter 34500 [Train]: loss=72.52, epe=5.12, lr=0.000065, samples/sec=40.6, sec/step=0.789, eta=3:23:43\n",
      "2018-09-08 13:47:02 Iter 34600 [Train]: loss=65.82, epe=4.60, lr=0.000066, samples/sec=40.7, sec/step=0.786, eta=3:21:47\n",
      "2018-09-08 13:48:30 Iter 34700 [Train]: loss=69.75, epe=4.91, lr=0.000068, samples/sec=40.7, sec/step=0.787, eta=3:20:36\n",
      "2018-09-08 13:49:59 Iter 34800 [Train]: loss=71.65, epe=5.07, lr=0.000069, samples/sec=40.6, sec/step=0.788, eta=3:19:42\n",
      "2018-09-08 13:51:28 Iter 34900 [Train]: loss=69.25, epe=4.87, lr=0.000070, samples/sec=40.9, sec/step=0.782, eta=3:16:49\n",
      "2018-09-08 13:52:56 Iter 35000 [Train]: loss=66.44, epe=4.63, lr=0.000071, samples/sec=40.7, sec/step=0.786, eta=3:16:33\n",
      "2018-09-08 13:53:12 Iter 35000 [Val]: loss=69.59, epe=4.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-35000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-35000\n",
      "2018-09-08 13:54:58 Iter 35100 [Train]: loss=71.33, epe=5.04, lr=0.000070, samples/sec=40.8, sec/step=0.785, eta=3:14:51\n",
      "2018-09-08 13:56:27 Iter 35200 [Train]: loss=70.71, epe=4.98, lr=0.000069, samples/sec=40.3, sec/step=0.794, eta=3:15:57\n",
      "2018-09-08 13:57:57 Iter 35300 [Train]: loss=66.23, epe=4.62, lr=0.000068, samples/sec=40.2, sec/step=0.796, eta=3:15:06\n",
      "2018-09-08 13:59:26 Iter 35400 [Train]: loss=72.63, epe=nan, lr=0.000066, samples/sec=40.5, sec/step=0.791, eta=3:12:23\n",
      "2018-09-08 14:00:55 Iter 35500 [Train]: loss=68.79, epe=4.85, lr=0.000065, samples/sec=40.5, sec/step=0.789, eta=3:10:44\n",
      "2018-09-08 14:02:25 Iter 35600 [Train]: loss=67.36, epe=4.72, lr=0.000064, samples/sec=40.2, sec/step=0.796, eta=3:11:05\n",
      "2018-09-08 14:03:54 Iter 35700 [Train]: loss=69.03, epe=4.84, lr=0.000063, samples/sec=40.3, sec/step=0.793, eta=3:09:01\n",
      "2018-09-08 14:05:23 Iter 35800 [Train]: loss=66.35, epe=4.64, lr=0.000061, samples/sec=40.8, sec/step=0.784, eta=3:05:26\n",
      "2018-09-08 14:06:53 Iter 35900 [Train]: loss=66.25, epe=4.64, lr=0.000060, samples/sec=40.2, sec/step=0.796, eta=3:07:09\n",
      "2018-09-08 14:08:22 Iter 36000 [Train]: loss=69.28, epe=4.87, lr=0.000059, samples/sec=40.3, sec/step=0.794, eta=3:05:14\n",
      "2018-09-08 14:08:38 Iter 36000 [Val]: loss=68.66, epe=4.87\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-36000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-36000\n",
      "2018-09-08 14:10:30 Iter 36100 [Train]: loss=69.52, epe=4.89, lr=0.000058, samples/sec=40.4, sec/step=0.791, eta=3:03:18\n",
      "2018-09-08 14:12:00 Iter 36200 [Train]: loss=65.93, epe=4.59, lr=0.000057, samples/sec=40.1, sec/step=0.798, eta=3:03:26\n",
      "2018-09-08 14:13:30 Iter 36300 [Train]: loss=66.62, epe=4.64, lr=0.000055, samples/sec=40.0, sec/step=0.800, eta=3:02:43\n",
      "2018-09-08 14:15:00 Iter 36400 [Train]: loss=69.42, epe=4.86, lr=0.000054, samples/sec=40.1, sec/step=0.799, eta=3:01:04\n",
      "2018-09-08 14:16:30 Iter 36500 [Train]: loss=65.13, epe=4.54, lr=0.000053, samples/sec=40.2, sec/step=0.797, eta=2:59:14\n",
      "2018-09-08 14:18:00 Iter 36600 [Train]: loss=69.73, epe=4.91, lr=0.000052, samples/sec=40.0, sec/step=0.799, eta=2:58:27\n",
      "2018-09-08 14:19:29 Iter 36700 [Train]: loss=71.31, epe=5.03, lr=0.000050, samples/sec=40.4, sec/step=0.793, eta=2:55:41\n",
      "2018-09-08 14:20:59 Iter 36800 [Train]: loss=71.34, epe=5.03, lr=0.000049, samples/sec=40.4, sec/step=0.792, eta=2:54:20\n",
      "2018-09-08 14:22:29 Iter 36900 [Train]: loss=65.31, epe=4.56, lr=0.000048, samples/sec=40.0, sec/step=0.799, eta=2:54:33\n",
      "2018-09-08 14:23:59 Iter 37000 [Train]: loss=67.54, epe=4.72, lr=0.000047, samples/sec=40.2, sec/step=0.797, eta=2:52:38\n",
      "2018-09-08 14:24:15 Iter 37000 [Val]: loss=67.95, epe=4.80\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-37000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-37000\n",
      "2018-09-08 14:26:02 Iter 37100 [Train]: loss=67.94, epe=4.74, lr=0.000046, samples/sec=40.4, sec/step=0.791, eta=2:50:06\n",
      "2018-09-08 14:27:32 Iter 37200 [Train]: loss=67.27, epe=4.68, lr=0.000044, samples/sec=40.0, sec/step=0.799, eta=2:50:28\n",
      "2018-09-08 14:29:02 Iter 37300 [Train]: loss=68.48, epe=4.81, lr=0.000043, samples/sec=39.7, sec/step=0.805, eta=2:50:27\n",
      "2018-09-08 14:30:33 Iter 37400 [Train]: loss=64.89, epe=4.50, lr=0.000042, samples/sec=40.0, sec/step=0.800, eta=2:47:56\n",
      "2018-09-08 14:32:03 Iter 37500 [Train]: loss=65.51, epe=4.57, lr=0.000041, samples/sec=39.7, sec/step=0.805, eta=2:47:47\n",
      "2018-09-08 14:33:34 Iter 37600 [Train]: loss=68.34, epe=4.77, lr=0.000039, samples/sec=39.7, sec/step=0.806, eta=2:46:38\n",
      "2018-09-08 14:35:05 Iter 37700 [Train]: loss=65.55, epe=4.55, lr=0.000038, samples/sec=39.8, sec/step=0.804, eta=2:44:50\n",
      "2018-09-08 14:36:35 Iter 37800 [Train]: loss=67.69, epe=4.74, lr=0.000037, samples/sec=39.7, sec/step=0.805, eta=2:43:46\n",
      "2018-09-08 14:38:06 Iter 37900 [Train]: loss=65.06, epe=4.52, lr=0.000036, samples/sec=39.8, sec/step=0.803, eta=2:41:58\n",
      "2018-09-08 14:39:36 Iter 38000 [Train]: loss=67.25, epe=4.70, lr=0.000035, samples/sec=40.0, sec/step=0.800, eta=2:39:55\n",
      "2018-09-08 14:39:52 Iter 38000 [Val]: loss=69.24, epe=4.89\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-38000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-38000\n",
      "2018-09-08 14:41:39 Iter 38100 [Train]: loss=68.67, epe=4.81, lr=0.000033, samples/sec=40.5, sec/step=0.791, eta=2:36:48\n",
      "2018-09-08 14:43:10 Iter 38200 [Train]: loss=64.92, epe=4.51, lr=0.000032, samples/sec=39.7, sec/step=0.806, eta=2:38:33\n",
      "2018-09-08 14:44:41 Iter 38300 [Train]: loss=64.11, epe=4.45, lr=0.000031, samples/sec=39.7, sec/step=0.807, eta=2:37:17\n",
      "2018-09-08 14:46:12 Iter 38400 [Train]: loss=70.34, epe=4.95, lr=0.000030, samples/sec=39.7, sec/step=0.806, eta=2:35:52\n",
      "2018-09-08 14:47:42 Iter 38500 [Train]: loss=66.23, epe=4.61, lr=0.000028, samples/sec=39.8, sec/step=0.803, eta=2:33:56\n",
      "2018-09-08 14:49:14 Iter 38600 [Train]: loss=68.33, epe=4.77, lr=0.000027, samples/sec=39.6, sec/step=0.809, eta=2:33:38\n",
      "2018-09-08 14:50:44 Iter 38700 [Train]: loss=66.70, epe=4.67, lr=0.000026, samples/sec=39.9, sec/step=0.802, eta=2:31:06\n",
      "2018-09-08 14:52:15 Iter 38800 [Train]: loss=64.30, epe=4.45, lr=0.000025, samples/sec=39.5, sec/step=0.810, eta=2:31:07\n",
      "2018-09-08 14:53:46 Iter 38900 [Train]: loss=64.95, epe=4.51, lr=0.000023, samples/sec=39.7, sec/step=0.806, eta=2:29:11\n",
      "2018-09-08 14:55:17 Iter 39000 [Train]: loss=67.53, epe=4.71, lr=0.000022, samples/sec=39.7, sec/step=0.806, eta=2:27:48\n",
      "2018-09-08 14:55:33 Iter 39000 [Val]: loss=69.09, epe=4.91\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-39000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-39000\n",
      "2018-09-08 14:57:21 Iter 39100 [Train]: loss=64.65, epe=4.49, lr=0.000021, samples/sec=40.0, sec/step=0.800, eta=2:25:22\n",
      "2018-09-08 14:58:53 Iter 39200 [Train]: loss=67.65, epe=4.73, lr=0.000020, samples/sec=39.5, sec/step=0.811, eta=2:25:56\n",
      "2018-09-08 15:00:24 Iter 39300 [Train]: loss=63.88, epe=4.43, lr=0.000019, samples/sec=39.6, sec/step=0.809, eta=2:24:14\n",
      "2018-09-08 15:01:55 Iter 39400 [Train]: loss=67.65, epe=4.73, lr=0.000017, samples/sec=39.5, sec/step=0.809, eta=2:22:58\n",
      "2018-09-08 15:03:26 Iter 39500 [Train]: loss=67.34, epe=4.70, lr=0.000016, samples/sec=39.4, sec/step=0.811, eta=2:21:58\n",
      "2018-09-08 15:04:57 Iter 39600 [Train]: loss=67.25, epe=4.68, lr=0.000015, samples/sec=39.7, sec/step=0.806, eta=2:19:43\n",
      "2018-09-08 15:06:28 Iter 39700 [Train]: loss=65.47, epe=4.55, lr=0.000014, samples/sec=39.6, sec/step=0.808, eta=2:18:40\n",
      "2018-09-08 15:07:59 Iter 39800 [Train]: loss=66.00, epe=4.61, lr=0.000012, samples/sec=39.5, sec/step=0.810, eta=2:17:41\n",
      "2018-09-08 15:09:30 Iter 39900 [Train]: loss=64.60, epe=4.48, lr=0.000011, samples/sec=39.6, sec/step=0.807, eta=2:15:53\n",
      "2018-09-08 15:11:01 Iter 40000 [Train]: loss=65.01, epe=4.50, lr=0.000010, samples/sec=39.5, sec/step=0.810, eta=2:14:56\n",
      "2018-09-08 15:11:18 Iter 40000 [Val]: loss=69.12, epe=4.90\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-40000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-40000\n",
      "2018-09-08 15:13:05 Iter 40100 [Train]: loss=66.04, epe=4.60, lr=0.000011, samples/sec=39.8, sec/step=0.805, eta=2:12:47\n",
      "2018-09-08 15:14:37 Iter 40200 [Train]: loss=66.26, epe=4.61, lr=0.000011, samples/sec=39.5, sec/step=0.810, eta=2:12:20\n",
      "2018-09-08 15:16:07 Iter 40300 [Train]: loss=64.66, epe=4.49, lr=0.000012, samples/sec=39.7, sec/step=0.806, eta=2:10:15\n",
      "2018-09-08 15:17:39 Iter 40400 [Train]: loss=65.61, epe=4.58, lr=0.000012, samples/sec=39.4, sec/step=0.812, eta=2:09:53\n",
      "2018-09-08 15:19:10 Iter 40500 [Train]: loss=65.15, epe=4.54, lr=0.000013, samples/sec=39.6, sec/step=0.809, eta=2:08:02\n",
      "2018-09-08 15:20:41 Iter 40600 [Train]: loss=65.22, epe=4.53, lr=0.000014, samples/sec=39.2, sec/step=0.816, eta=2:07:52\n",
      "2018-09-08 15:22:12 Iter 40700 [Train]: loss=69.92, epe=4.91, lr=0.000014, samples/sec=39.6, sec/step=0.808, eta=2:05:14\n",
      "2018-09-08 15:23:43 Iter 40800 [Train]: loss=66.51, epe=4.61, lr=0.000015, samples/sec=39.6, sec/step=0.808, eta=2:03:53\n",
      "2018-09-08 15:25:14 Iter 40900 [Train]: loss=65.64, epe=4.56, lr=0.000016, samples/sec=39.6, sec/step=0.807, eta=2:02:25\n",
      "2018-09-08 15:26:45 Iter 41000 [Train]: loss=62.62, epe=4.33, lr=0.000016, samples/sec=39.8, sec/step=0.804, eta=2:00:39\n",
      "2018-09-08 15:27:02 Iter 41000 [Val]: loss=67.45, epe=4.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-41000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-41000\n",
      "2018-09-08 15:28:49 Iter 41100 [Train]: loss=64.04, epe=4.45, lr=0.000017, samples/sec=40.0, sec/step=0.801, eta=1:58:45\n",
      "2018-09-08 15:30:20 Iter 41200 [Train]: loss=65.09, epe=4.53, lr=0.000017, samples/sec=39.8, sec/step=0.804, eta=1:57:55\n",
      "2018-09-08 15:31:51 Iter 41300 [Train]: loss=66.46, epe=4.63, lr=0.000018, samples/sec=39.6, sec/step=0.808, eta=1:57:11\n",
      "2018-09-08 15:33:22 Iter 41400 [Train]: loss=65.14, epe=4.53, lr=0.000019, samples/sec=39.4, sec/step=0.811, eta=1:56:19\n",
      "2018-09-08 15:34:53 Iter 41500 [Train]: loss=66.63, epe=4.63, lr=0.000019, samples/sec=39.5, sec/step=0.810, eta=1:54:44\n",
      "2018-09-08 15:36:24 Iter 41600 [Train]: loss=63.49, epe=4.38, lr=0.000020, samples/sec=39.6, sec/step=0.808, eta=1:53:07\n",
      "2018-09-08 15:37:55 Iter 41700 [Train]: loss=65.08, epe=4.53, lr=0.000020, samples/sec=39.3, sec/step=0.813, eta=1:52:31\n",
      "2018-09-08 15:39:26 Iter 41800 [Train]: loss=66.73, epe=4.65, lr=0.000021, samples/sec=39.6, sec/step=0.809, eta=1:50:31\n",
      "2018-09-08 15:40:58 Iter 41900 [Train]: loss=65.89, epe=4.58, lr=0.000022, samples/sec=39.5, sec/step=0.810, eta=1:49:18\n",
      "2018-09-08 15:42:29 Iter 42000 [Train]: loss=65.90, epe=4.59, lr=0.000022, samples/sec=39.6, sec/step=0.807, eta=1:47:40\n",
      "2018-09-08 15:42:45 Iter 42000 [Val]: loss=68.01, epe=4.81\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-42000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-42000\n",
      "2018-09-08 15:44:33 Iter 42100 [Train]: loss=68.82, epe=4.78, lr=0.000023, samples/sec=40.0, sec/step=0.801, eta=1:45:25\n",
      "2018-09-08 15:46:04 Iter 42200 [Train]: loss=65.64, epe=4.56, lr=0.000023, samples/sec=39.5, sec/step=0.810, eta=1:45:14\n",
      "2018-09-08 15:47:35 Iter 42300 [Train]: loss=63.27, epe=4.38, lr=0.000024, samples/sec=39.7, sec/step=0.806, eta=1:43:27\n",
      "2018-09-08 15:49:06 Iter 42400 [Train]: loss=68.85, epe=4.83, lr=0.000025, samples/sec=39.4, sec/step=0.811, eta=1:42:46\n",
      "2018-09-08 15:50:38 Iter 42500 [Train]: loss=65.26, epe=4.55, lr=0.000025, samples/sec=39.3, sec/step=0.813, eta=1:41:40\n",
      "2018-09-08 15:52:09 Iter 42600 [Train]: loss=64.47, epe=4.47, lr=0.000026, samples/sec=39.4, sec/step=0.812, eta=1:40:09\n",
      "2018-09-08 15:53:40 Iter 42700 [Train]: loss=66.32, epe=4.62, lr=0.000027, samples/sec=39.4, sec/step=0.812, eta=1:38:47\n",
      "2018-09-08 15:55:12 Iter 42800 [Train]: loss=65.83, epe=4.57, lr=0.000027, samples/sec=39.5, sec/step=0.811, eta=1:37:20\n",
      "2018-09-08 15:56:43 Iter 42900 [Train]: loss=64.64, epe=4.49, lr=0.000028, samples/sec=39.2, sec/step=0.817, eta=1:36:43\n",
      "2018-09-08 15:58:15 Iter 43000 [Train]: loss=64.12, epe=4.45, lr=0.000028, samples/sec=39.3, sec/step=0.814, eta=1:35:01\n",
      "2018-09-08 15:58:32 Iter 43000 [Val]: loss=67.11, epe=4.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-43000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-43000\n",
      "2018-09-08 16:00:19 Iter 43100 [Train]: loss=67.45, epe=4.70, lr=0.000029, samples/sec=39.7, sec/step=0.806, eta=1:32:39\n",
      "2018-09-08 16:01:51 Iter 43200 [Train]: loss=63.47, epe=4.40, lr=0.000030, samples/sec=39.2, sec/step=0.817, eta=1:32:37\n",
      "2018-09-08 16:03:23 Iter 43300 [Train]: loss=66.12, epe=4.60, lr=0.000030, samples/sec=39.2, sec/step=0.817, eta=1:31:13\n",
      "2018-09-08 16:04:55 Iter 43400 [Train]: loss=64.38, epe=4.47, lr=0.000031, samples/sec=39.4, sec/step=0.811, eta=1:29:15\n",
      "2018-09-08 16:06:26 Iter 43500 [Train]: loss=64.85, epe=4.50, lr=0.000031, samples/sec=39.4, sec/step=0.812, eta=1:28:00\n",
      "2018-09-08 16:07:58 Iter 43600 [Train]: loss=67.79, epe=4.73, lr=0.000032, samples/sec=39.4, sec/step=0.812, eta=1:26:38\n",
      "2018-09-08 16:09:30 Iter 43700 [Train]: loss=67.50, epe=4.72, lr=0.000033, samples/sec=39.3, sec/step=0.815, eta=1:25:34\n",
      "2018-09-08 16:11:02 Iter 43800 [Train]: loss=67.34, epe=4.69, lr=0.000033, samples/sec=39.1, sec/step=0.818, eta=1:24:32\n",
      "2018-09-08 16:12:34 Iter 43900 [Train]: loss=64.36, epe=4.47, lr=0.000034, samples/sec=39.3, sec/step=0.815, eta=1:22:52\n",
      "2018-09-08 16:14:06 Iter 44000 [Train]: loss=65.38, epe=4.54, lr=0.000035, samples/sec=38.9, sec/step=0.822, eta=1:22:13\n",
      "2018-09-08 16:14:23 Iter 44000 [Val]: loss=67.10, epe=4.74\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-44000 is not in all_model_checkpoint_paths. Manually adding it.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-44000\n",
      "2018-09-08 16:16:12 Iter 44100 [Train]: loss=68.37, epe=4.80, lr=0.000035, samples/sec=39.2, sec/step=0.815, eta=1:20:11\n",
      "2018-09-08 16:17:44 Iter 44200 [Train]: loss=64.01, epe=4.44, lr=0.000036, samples/sec=39.3, sec/step=0.815, eta=1:18:48\n",
      "2018-09-08 16:19:16 Iter 44300 [Train]: loss=66.37, epe=4.64, lr=0.000036, samples/sec=39.1, sec/step=0.818, eta=1:17:45\n",
      "2018-09-08 16:20:48 Iter 44400 [Train]: loss=66.94, epe=4.66, lr=0.000037, samples/sec=39.4, sec/step=0.811, eta=1:15:44\n",
      "2018-09-08 16:22:20 Iter 44500 [Train]: loss=65.17, epe=4.54, lr=0.000038, samples/sec=39.1, sec/step=0.817, eta=1:14:56\n",
      "2018-09-08 16:23:52 Iter 44600 [Train]: loss=65.51, epe=4.57, lr=0.000038, samples/sec=39.0, sec/step=0.821, eta=1:13:53\n",
      "2018-09-08 16:25:24 Iter 44700 [Train]: loss=64.19, epe=4.44, lr=0.000039, samples/sec=39.3, sec/step=0.814, eta=1:11:54\n",
      "2018-09-08 16:26:56 Iter 44800 [Train]: loss=67.68, epe=4.73, lr=0.000039, samples/sec=39.3, sec/step=0.814, eta=1:10:31\n",
      "2018-09-08 16:28:28 Iter 44900 [Train]: loss=68.39, epe=4.79, lr=0.000040, samples/sec=39.1, sec/step=0.817, eta=1:09:29\n",
      "2018-09-08 16:29:59 Iter 45000 [Train]: loss=64.14, epe=4.44, lr=0.000041, samples/sec=39.3, sec/step=0.815, eta=1:07:54\n",
      "2018-09-08 16:30:16 Iter 45000 [Val]: loss=65.04, epe=4.55\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-45000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-45000\n",
      "2018-09-08 16:32:11 Iter 45100 [Train]: loss=65.31, epe=4.55, lr=0.000040, samples/sec=39.3, sec/step=0.814, eta=1:06:30\n",
      "2018-09-08 16:33:44 Iter 45200 [Train]: loss=65.50, epe=4.55, lr=0.000039, samples/sec=38.8, sec/step=0.824, eta=1:05:56\n",
      "2018-09-08 16:35:17 Iter 45300 [Train]: loss=63.68, epe=4.39, lr=0.000039, samples/sec=38.8, sec/step=0.825, eta=1:04:40\n",
      "2018-09-08 16:36:50 Iter 45400 [Train]: loss=65.89, epe=4.59, lr=0.000038, samples/sec=38.6, sec/step=0.829, eta=1:03:33\n",
      "2018-09-08 16:38:23 Iter 45500 [Train]: loss=66.73, epe=4.65, lr=0.000038, samples/sec=38.9, sec/step=0.823, eta=1:01:43\n",
      "2018-09-08 16:39:55 Iter 45600 [Train]: loss=64.19, epe=4.46, lr=0.000037, samples/sec=39.1, sec/step=0.818, eta=0:59:58\n",
      "2018-09-08 16:41:27 Iter 45700 [Train]: loss=64.33, epe=4.46, lr=0.000036, samples/sec=38.8, sec/step=0.824, eta=0:59:03\n",
      "2018-09-08 16:43:00 Iter 45800 [Train]: loss=69.08, epe=4.85, lr=0.000036, samples/sec=39.0, sec/step=0.821, eta=0:57:27\n",
      "2018-09-08 16:44:32 Iter 45900 [Train]: loss=66.53, epe=4.63, lr=0.000035, samples/sec=39.2, sec/step=0.817, eta=0:55:48\n",
      "2018-09-08 16:46:05 Iter 46000 [Train]: loss=69.54, epe=4.87, lr=0.000035, samples/sec=38.7, sec/step=0.827, eta=0:55:09\n",
      "2018-09-08 16:46:22 Iter 46000 [Val]: loss=68.05, epe=4.77\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-46000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-46000\n",
      "2018-09-08 16:48:11 Iter 46100 [Train]: loss=67.82, epe=4.72, lr=0.000034, samples/sec=39.1, sec/step=0.819, eta=0:53:15\n",
      "2018-09-08 16:49:44 Iter 46200 [Train]: loss=65.46, epe=4.55, lr=0.000033, samples/sec=38.8, sec/step=0.824, eta=0:52:10\n",
      "2018-09-08 16:51:18 Iter 46300 [Train]: loss=65.45, epe=4.55, lr=0.000033, samples/sec=38.5, sec/step=0.832, eta=0:51:19\n",
      "2018-09-08 16:52:51 Iter 46400 [Train]: loss=63.75, epe=4.42, lr=0.000032, samples/sec=38.7, sec/step=0.826, eta=0:49:34\n",
      "2018-09-08 16:54:24 Iter 46500 [Train]: loss=66.64, epe=4.64, lr=0.000031, samples/sec=38.7, sec/step=0.827, eta=0:48:13\n",
      "2018-09-08 16:55:57 Iter 46600 [Train]: loss=63.98, epe=4.44, lr=0.000031, samples/sec=38.8, sec/step=0.825, eta=0:46:45\n",
      "2018-09-08 16:57:29 Iter 46700 [Train]: loss=66.16, epe=4.60, lr=0.000030, samples/sec=38.7, sec/step=0.826, eta=0:45:27\n",
      "2018-09-08 16:59:02 Iter 46800 [Train]: loss=62.43, epe=4.32, lr=0.000030, samples/sec=38.8, sec/step=0.824, eta=0:43:57\n",
      "2018-09-08 17:00:35 Iter 46900 [Train]: loss=62.99, epe=4.35, lr=0.000029, samples/sec=38.8, sec/step=0.826, eta=0:42:40\n",
      "2018-09-08 17:02:08 Iter 47000 [Train]: loss=65.65, epe=4.56, lr=0.000028, samples/sec=38.6, sec/step=0.830, eta=0:41:30\n",
      "2018-09-08 17:02:25 Iter 47000 [Val]: loss=70.03, epe=4.97\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (4.97) doesn't outperform other checkpoints\n",
      "2018-09-08 17:04:10 Iter 47100 [Train]: loss=69.16, epe=4.85, lr=0.000028, samples/sec=39.5, sec/step=0.811, eta=0:39:10\n",
      "2018-09-08 17:05:43 Iter 47200 [Train]: loss=67.72, epe=4.75, lr=0.000027, samples/sec=38.7, sec/step=0.826, eta=0:38:34\n",
      "2018-09-08 17:07:15 Iter 47300 [Train]: loss=63.35, epe=4.39, lr=0.000027, samples/sec=38.9, sec/step=0.823, eta=0:37:03\n",
      "2018-09-08 17:08:48 Iter 47400 [Train]: loss=65.79, epe=4.58, lr=0.000026, samples/sec=39.0, sec/step=0.820, eta=0:35:32\n",
      "2018-09-08 17:10:21 Iter 47500 [Train]: loss=64.29, epe=4.44, lr=0.000025, samples/sec=38.8, sec/step=0.825, eta=0:34:23\n",
      "2018-09-08 17:11:52 Iter 47600 [Train]: loss=63.63, epe=4.41, lr=0.000025, samples/sec=39.5, sec/step=0.811, eta=0:32:25\n",
      "2018-09-08 17:13:24 Iter 47700 [Train]: loss=65.39, epe=4.53, lr=0.000024, samples/sec=39.2, sec/step=0.817, eta=0:31:18\n",
      "2018-09-08 17:14:57 Iter 47800 [Train]: loss=65.20, epe=4.53, lr=0.000023, samples/sec=38.9, sec/step=0.823, eta=0:30:11\n",
      "2018-09-08 17:16:29 Iter 47900 [Train]: loss=65.41, epe=4.54, lr=0.000023, samples/sec=38.8, sec/step=0.824, eta=0:28:51\n",
      "2018-09-08 17:18:02 Iter 48000 [Train]: loss=66.48, epe=4.64, lr=0.000022, samples/sec=38.9, sec/step=0.823, eta=0:27:27\n",
      "2018-09-08 17:18:19 Iter 48000 [Val]: loss=66.86, epe=4.68\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-48000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-48000\n",
      "2018-09-08 17:20:08 Iter 48100 [Train]: loss=63.56, epe=4.39, lr=0.000022, samples/sec=38.8, sec/step=0.824, eta=0:26:05\n",
      "2018-09-08 17:21:42 Iter 48200 [Train]: loss=65.91, epe=4.58, lr=0.000021, samples/sec=38.6, sec/step=0.829, eta=0:24:52\n",
      "2018-09-08 17:23:15 Iter 48300 [Train]: loss=62.41, epe=4.31, lr=0.000020, samples/sec=38.6, sec/step=0.829, eta=0:23:30\n",
      "2018-09-08 17:24:49 Iter 48400 [Train]: loss=66.11, epe=4.59, lr=0.000020, samples/sec=38.5, sec/step=0.832, eta=0:22:11\n",
      "2018-09-08 17:26:22 Iter 48500 [Train]: loss=66.55, epe=4.64, lr=0.000019, samples/sec=38.5, sec/step=0.832, eta=0:20:48\n",
      "2018-09-08 17:27:56 Iter 48600 [Train]: loss=64.57, epe=4.47, lr=0.000019, samples/sec=38.5, sec/step=0.832, eta=0:19:25\n",
      "2018-09-08 17:29:29 Iter 48700 [Train]: loss=66.53, epe=4.62, lr=0.000018, samples/sec=38.6, sec/step=0.829, eta=0:17:58\n",
      "2018-09-08 17:31:03 Iter 48800 [Train]: loss=63.25, epe=4.37, lr=0.000017, samples/sec=38.3, sec/step=0.836, eta=0:16:43\n",
      "2018-09-08 17:32:37 Iter 48900 [Train]: loss=62.24, epe=4.30, lr=0.000017, samples/sec=38.5, sec/step=0.831, eta=0:15:14\n",
      "2018-09-08 17:34:10 Iter 49000 [Train]: loss=64.79, epe=4.51, lr=0.000016, samples/sec=38.5, sec/step=0.831, eta=0:13:51\n",
      "2018-09-08 17:34:28 Iter 49000 [Val]: loss=63.67, epe=4.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-49000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-49000\n",
      "2018-09-08 17:36:18 Iter 49100 [Train]: loss=61.91, epe=4.27, lr=0.000016, samples/sec=38.5, sec/step=0.831, eta=0:12:28\n",
      "2018-09-08 17:37:52 Iter 49200 [Train]: loss=64.64, epe=4.47, lr=0.000015, samples/sec=38.4, sec/step=0.834, eta=0:11:07\n",
      "2018-09-08 17:39:25 Iter 49300 [Train]: loss=65.16, epe=4.50, lr=0.000014, samples/sec=38.5, sec/step=0.832, eta=0:09:42\n",
      "2018-09-08 17:41:00 Iter 49400 [Train]: loss=63.85, epe=4.42, lr=0.000014, samples/sec=38.1, sec/step=0.840, eta=0:08:24\n",
      "2018-09-08 17:42:33 Iter 49500 [Train]: loss=63.75, epe=4.40, lr=0.000013, samples/sec=38.4, sec/step=0.833, eta=0:06:56\n",
      "2018-09-08 17:44:07 Iter 49600 [Train]: loss=64.96, epe=4.50, lr=0.000012, samples/sec=38.3, sec/step=0.835, eta=0:05:34\n",
      "2018-09-08 17:45:41 Iter 49700 [Train]: loss=64.32, epe=4.44, lr=0.000012, samples/sec=38.2, sec/step=0.838, eta=0:04:11\n",
      "2018-09-08 17:47:15 Iter 49800 [Train]: loss=64.98, epe=4.50, lr=0.000011, samples/sec=38.3, sec/step=0.836, eta=0:02:47\n",
      "2018-09-08 17:48:49 Iter 49900 [Train]: loss=64.75, epe=4.49, lr=0.000011, samples/sec=38.4, sec/step=0.834, eta=0:01:23\n",
      "2018-09-08 17:50:23 Iter 50000 [Train]: loss=64.18, epe=4.44, lr=0.000010, samples/sec=38.1, sec/step=0.839, eta=0:00:00\n",
      "2018-09-08 17:50:41 Iter 50000 [Val]: loss=66.57, epe=4.67\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-50000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-sm-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-50000\n",
      "... done training.\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "nn.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the training curves for the run above:\n",
    "\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/loss.png)\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/epe.png)\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/lr.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the predictions issued by the model for a few validation samples:\n",
    "\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/val1.png)\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/val2.png)\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/val3.png)\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/val4.png)\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/val5.png)\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/val6.png)\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/val7.png)\n",
    "![](img/pwcnet-sm-6-2-cyclic-chairsthingsmix/val8.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}