devkit_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ecc7364b",
   "metadata": {},
   "source": [
    "## Development-Kit Tutorial for Zenseact Open Dataset\n",
    "\n",
    "Welcome to this tutorial on using the development kit of the Zenseact Open dataset.\n",
    "This notebook introduces us to working with the data loaders, coordinate transformation, and visualization functionalities. Data loaders are provided for camera images, LiDAR point clouds, high-precision GPS (a.k.a. OXTS) sequences, vehicle data, calibration files, and different annotation tasks, in addition to some detailed descriptions about the data. More details can be found in the dataset description document.\n",
    "\n",
    "Make sure you have set DATA_ROOT variable in the constants.py to point to the right path for the dataset before running the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc26feb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from datetime import datetime\n",
    "from matplotlib import pyplot as plt\n",
    "# ensures that the graphs are displayed in the notebook along with the code\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = [20, 10]\n",
    "import json\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import cv2\n",
    "\n",
    "import data_utils\n",
    "\n",
    "from plot_gps import plot_gps_track_from_dataset_sequence\n",
    "from plot_gps_on_image import visualize_gps_on_image\n",
    "from plot_lidar_on_image import visualize_lidar_on_image\n",
    "from plot_objects_annot_on_image import visualize_annotated_objects_on_image\n",
    "from plot_polygons_annot_on_image import visualize_annotated_polygons_on_image\n",
    "from calibration import load_calib_from_json\n",
    "from constants import (PNG_EXT, TIME_FORMAT, DYNAMIC_OBJECTS, LANE_MARKINGS, STATIC_OBJECTS,\n",
    "                       EGO_ROAD, BLURRED_IMAGES, DATA_ROOT)\n",
    "from plot_lidar_bev import BEVBox\n",
    "from bev_utils import get_objects_for_bev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47579cc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "ANNOTATION_FOLDER = os.path.join(DATA_ROOT, \"annotations\")\n",
    "OXTS_FOLDER = os.path.join(DATA_ROOT, \"oxts_data\")\n",
    "VISION_FOLDER = os.path.join(DATA_ROOT, BLURRED_IMAGES)\n",
    "CALIBRATIONS_FOLDER = os.path.join(DATA_ROOT, \"calibration\")\n",
    "LIDAR_FOLDER = os.path.join(DATA_ROOT, \"lidar_data\")\n",
    "RANGE_LIDAR_FOLDER = os.path.join(DATA_ROOT, \"range_lidar_data\")\n",
    "VDATA_FOLDER = os.path.join(DATA_ROOT, \"vehicle_data\")\n",
    "SEQUENCE_FOLDER = \"93_2021-04-18T16:04:33.891575Z\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d124aa5",
   "metadata": {},
   "source": [
    "Load and visualize the blurred camera image for the core/middle frame in the sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b451d410",
   "metadata": {},
   "outputs": [],
   "source": [
    "INDEX = 1\n",
    "IMAGES, IMAGE_FILES = data_utils.load_images_from_dataset(os.path.join(VISION_FOLDER, SEQUENCE_FOLDER))\n",
    "IMAGE = IMAGES[INDEX]\n",
    "IMAGE_FILE = IMAGE_FILES[INDEX]\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(cv2.cvtColor(IMAGE, cv2.COLOR_BGR2RGB))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0c8242d",
   "metadata": {},
   "source": [
    "Load the calibration file for the given sequence. Calibration files are provided per date, so we need to get the frame timestamp and the vehicle name first from the file name to load the proper calibration file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f47a1a5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get info from the file\n",
    "vehicle, camera_name, time_str, sequence = os.path.basename(IMAGE_FILE.replace(PNG_EXT[1:], \"\")).split(\"_\")\n",
    "frame_time = datetime.strptime(time_str, TIME_FORMAT)\n",
    "\n",
    "# get proper calibration file from the calibration folder in the dataset \n",
    "calib = load_calib_from_json(CALIBRATIONS_FOLDER, vehicle, frame_time, camera_name)\n",
    "calib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5adeddc",
   "metadata": {},
   "source": [
    "Load high-precision GPS (a.k.a OXTS) for the given sequence. The following table describes the OXTS fields.\n",
    "\n",
    "OXTS data is provided in [-1s, ~10s] around the core frames for each sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba594779",
   "metadata": {},
   "outputs": [],
   "source": [
    "OXTS_DATA = data_utils.load_oxts_from_dataset(os.path.join(OXTS_FOLDER, SEQUENCE_FOLDER))\n",
    "OXTS_DATA.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b802f235",
   "metadata": {},
   "source": [
    "# **OxTS fields description:**\n",
    "\n",
    "| Name | Type | Units | Description |\n",
    "| --- | --- | --- | --- |\n",
    "| 'timestamp' | double |  seconds  | seconds\tUTC timestamp of each pose. |\n",
    "|  'datumEllipsoid'  |  string  |    |  ellipsoid model used (WGS84 or GRS80)  |\n",
    "|  'undulation'  |  double  |    |  Undulation (difference between oxts unit latitude and WGW-84 ellipsoidal altitude)  |\n",
    "|  'earthFrame'  |  string  |    |  earth frame associated with datum ellipsoid (ITRF2008, ETRF2000 or NAD83)  |\n",
    "|  'posLat'  |  double  |  degrees  |  degrees\tWGS84 Latitude  |\n",
    "|  'posLon'  |  double  |  degrees  |  degrees\tWGS84 Longitude  |\n",
    "|  'posAlt'  |  double  |  meters  |  meters\tWGS84 Altitude  |\n",
    "|  'velNorth'  |  double  |  m/sec  |  North component of velocity vector  |\n",
    "|  'velEast'  |  double  |  m/sec  |  East component of velocity vector  |\n",
    "|  'velDown'  |  double  |  m/sec  |  Vertical (down) component of velocity vector  |\n",
    "|  'velForward'  |  double  |  m/sec  |  Forward component of vehicle velocity vector (ahead, parallel to ground plane)  |\n",
    "|  'velLateral'  |  double  |  m/sec  |  Lateral component of velocity vector (to left, parallel to ground plane).  |\n",
    "|  'heading'  |  double  |  deegrees  |  Vehicle heading (clockwise from North in top view. Could be of (-180, +180).  |\n",
    "|  'pitch'  |  double  |  degrees  |  Vehicle pitch (counterclockwise from horizontal in right view). Could be [-90, +90] 0 when vehicle X axis is horizontal, front to up increases pitch.  |\n",
    "|  'roll'  |  double  |  degrees  |  Vehicle roll (counterclockwise from horizontal in back view) could be of (-180, +180).0 when vehicle Y axis is horizontal, left side down increases roll.  |\n",
    "|  'slipAngle'  |  double  |  degrees  |  Shows vehicle skidding, left side to front makes positive slipAngle. If vehicle forward velocity less then 3 m/sec, slipAngle assumed as 0  |\n",
    "|  'satellites'  |    |    |  number of satellites used  |\n",
    "|  'positionMode'  |  single  |    |  value ranges form 0-27. Explanation in table9 in OXTS/manuals/ncomman.pdf  |\n",
    "|  'velocityMode'  |  single  |    |  value ranges form 0-27. Explanation in table9 in OXTS/manuals/ncomman.pdf  |\n",
    "|  'orientationMode'  |  single  |    |  value ranges form 0-27. Explanation in table9 in OXTS/manuals/ncomman.pdf  |\n",
    "|  'stdDevPosNorth'  |  double  |    |  standard deviation for PosLocalNorth value  |\n",
    "|  'stdDevPosEast'  |  double  |    |  standard deviation for PosLocalEarth value  |\n",
    "|  'stdDevPosDown'  |  double  |    |  standard deviation for posNorth value  |\n",
    "|  'stdDevVelNorth'  |  double  |    |  standard deviation for velNorth value  |\n",
    "|  'stdDevVelEast'  |  double  |    |  standard deviation for velEast value  |\n",
    "|  'stdDevVelDown'  |  double  |    |  standard deviation for velDown value  |\n",
    "|  'stdDevHeading'  |  double  |    |  standard deviation for heading value  |\n",
    "|  'stdDevPitch'  |  double  |    |  standard deviation for pitch value  |\n",
    "|  'stdDevRoll'  |  double  |    |  standard deviation for roll value  |\n",
    "|  'accelerationX'  |  double  |  m/sec2  |  X component of the vehicle acceleration vector (forward positive)  |\n",
    "|  'accelerationY'  |  double  |  m/sec2  |  Y component of the vehicle acceleration vector (to left positive)  |\n",
    "|  'accelerationZ'  |  double  |  m/sec2  |  Z component of the vehicle acceleration vector (to up positive)  |\n",
    "|  'accelerometerBias'  |    |    |  offset of its output signal from the actual acceleration value  |\n",
    "|  'angularRateX'  |  double  |  deg/sec  |  Angular acceleration around vehicle x (forward) axis. Counterclockwise like roll  |\n",
    "|  'angularRateY'  |  double  |  deg/sec  |  Angular acceleration around vehicle Y (to left) axis.  Counterclockwise like pitch  |\n",
    "|  'angularRateZ'  |  double  |  deg/sec  |  Angular acceleration around vehicle Z (vertical) axis. Clockwise like heading  |\n",
    "|  'gyroBias'  |  double  |  radians/seconds  |  offset of gyroscope output signal from the actual value  |\n",
    "|  'configurationMisalignment'  |    |    |    |\n",
    "|  'leapSeconds'  |    |    |    |\n",
    "|  'speed'  |  double  |  m/sec  |  Vehicle speed vector length.  |\n",
    "|  'isValid'  |  uint8  |    |  1 if at least one, oxts or RoadRunner, has valid pose of corresponding timestamp. 0 otherwise.  |\n",
    "|  'isValidXY'  |  uint8  |    |    |\n",
    "|  'isValidHeading'  | uint8   |    |  1 if heading value is valid, 0 otherwise  |\n",
    "|  'isValidNorthVelocity'  |  uint8  |    |  1 if velNorth value is valid, 0 otherwise  |\n",
    "|  'isValidEastVelocity'  |  uint8  |    |  1 if velEast value is valid, 0 otherwise  |\n",
    "|  'isValidAltitude'  |  uint8  |    |  1 if posAlt value is valid, 0 otherwise  |\n",
    "|  'isValidLatLong'  |  uint8  |    |  1 if posLat and posLon values are valid, 0 otherwise  |\n",
    "|  'isValidFwdBwd'  |  uint8  |    |    |\n",
    "|  'fwdBwdError'  |    |    |  error of forward/backward processing  |\n",
    "|  'estPosError'  |    |    |    |\n",
    "|  'accuracyBiasX'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.   |\n",
    "|  'accuracyBiasY'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  |\n",
    "|  'accuracyBiasZ'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  |\n",
    "|  'BiasX'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  |\n",
    "|  'BiasY'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  |\n",
    "|  'BiasZ'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  |\n",
    "|  'scaleFactorX'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  |\n",
    "|  'scaleFactorY'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  |\n",
    "|  'scaleFactorZ'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  |\n",
    "|  'accuracyBiasX'  |    |    |    |\n",
    "|  'accuracyBiasY'  |    |    |    |\n",
    "|  'accuracyBiasZ'  |    |    |    |\n",
    "|  'accuracyScaleFactorX'  |    |    |    |\n",
    "|  'accuracyScaleFactorY'  |    |    |    |\n",
    "|  'accuracyScaleFactorZ'  |    |    |    |\n",
    "|  'headingAccuracy'  |    |    |    |\n",
    "|  'pitchAccuracy'  |    |    |    |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebea1673",
   "metadata": {},
   "outputs": [],
   "source": [
    "# show content of OXTS file\n",
    "OXTS_DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfccad83",
   "metadata": {},
   "source": [
    "Visualize OXTS trajectory on the interactive map of Warsaw and project it over the core camera image.\n",
    "\n",
    "The projected OXTS trajectory shows how the ego vehicle is driven 200 meters ahead. To do so, OXTS data is transformed into the camera coordinate system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83ed5196",
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot GPS track on interactive map\n",
    "plot_gps_track_from_dataset_sequence(OXTS_DATA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "473c3d70",
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize GPS track over image\n",
    "image = visualize_gps_on_image(OXTS_DATA, frame_time, calib, IMAGE)\n",
    "image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "plt.imshow(image)\n",
    "plt.title('GPS on image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58970d25",
   "metadata": {},
   "source": [
    "Load single frame LiDAR point cloud for the given sequence. The LiDAR point cloud is the closest LiDAR scan to the camera timestamp of the core frame.\n",
    "\n",
    "Zenseact Open Dataset also provides a range of LiDAR point clouds captured in [-1s, +1s] at 10Hz around the core frame for the sequences.\n",
    "The same data loaders can load LiDAR point clouds from the range_lidar_data. See the given example below.\n",
    "\n",
    "A description of the LiDAR point cloud fields can be found in the following table.\n",
    "\n",
    "It is worth mentioning that a few sequences have less than 21 LiDAR scans in the range_lidar_data (1 with 15 scans, 3 with 19 scans, and 74 with 20 scans), since these sequences were at the beginning or end of the logged data, and no information is available from before or after."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "649f8e52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load lidar data for frame sequence\n",
    "LIDAR_POINTCLOUD, LIDAR_FILES = data_utils.load_lidar_from_dataset(os.path.join(LIDAR_FOLDER, SEQUENCE_FOLDER))\n",
    "print(LIDAR_FILES)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0febf5f7",
   "metadata": {},
   "source": [
    "# **Lidar fields description:**\n",
    "\n",
    "| Name | Type | Units | Description |\n",
    "| --- | --- | --- | --- |\n",
    "| 'timestamp' | string |  seconds  | UTC timestamp of each point. |\n",
    "| 'x' | double |  meters  | x coordinate of the point in lidar frame |\n",
    "| 'y' | double |  meters  | y coordinate of the point in lidar frame |\n",
    "| 'z' | double |  meters  | z coordinate of the point in lidar frame |\n",
    "| 'intensity' | double |    | intensity level of each point in range [0..255] |\n",
    "| 'diode_index' | integer |    | index of diode emitter which produced a point (1..128) |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cbca724",
   "metadata": {},
   "source": [
    "In the following, LiDAT point clouds are projected into the camera coordinate system and overlaid on the image. Color of the points represent their normalized depth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74d7ab62",
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize lidar points projected over image\n",
    "image = visualize_lidar_on_image(LIDAR_POINTCLOUD[:, :3], calib, IMAGE)\n",
    "image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "plt.imshow(image)\n",
    "plt.title('Lidar on image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c60e4679",
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize lidar points of lidar scan 1s before image timestamp projected over image\n",
    "PREV_LIDAR_POINTCLOUD, LIDAR_FILE = data_utils.load_lidar_from_dataset(\n",
    "    os.path.join(RANGE_LIDAR_FOLDER, SEQUENCE_FOLDER),\n",
    "    index=0\n",
    ")\n",
    "\n",
    "image = visualize_lidar_on_image(PREV_LIDAR_POINTCLOUD[:, :3], calib, IMAGE)\n",
    "image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "plt.imshow(image)\n",
    "plt.title('Lidar on image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f942af0c",
   "metadata": {},
   "source": [
    "Load vehicle data for the given sequence, and show its content. Vehicle data covers [-1s, +1s] data around the core frames. Note that vehicle data is missing for a very few of the sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fc70aa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "VEHICLE_DATA = data_utils.load_vehicle_data_from_dataset(os.path.join(VDATA_FOLDER, SEQUENCE_FOLDER))\n",
    "VEHICLE_DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f5a1b28",
   "metadata": {},
   "source": [
    "#### There are 4 types of annotationed objects:  \n",
    "1. **dynamic_objects** - objects that can move (vehicles, pedestrians etc.) - annotated with 2D/3D bounding boxes\n",
    "2. **static_objects** - non-movable objects (light poles, traffic signs etc.) - annotated with 2D/3D bounding boxes\n",
    "3. **lane_markings** - lane markings and road paitings - annotated with polygons\n",
    "4. **ego_road** - polygons that shows the road where ego vehicle can drive - annotated with polygons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94a11f39",
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_TO_PROPERTIES = {\n",
    "    DYNAMIC_OBJECTS: [\"class\", \"occlusion_ratio\", \"object_type\"],\n",
    "    LANE_MARKINGS: [\"class\", \"coloured\"],\n",
    "    STATIC_OBJECTS: [\"class\", \"occlusion_ratio\", \"is_for_construction\"],\n",
    "    EGO_ROAD: [\"class\"],\n",
    "}\n",
    "\n",
    "SEQ_FOLDER = os.path.join(DATA_ROOT, \"{}/\", SEQUENCE_FOLDER)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09441d40",
   "metadata": {},
   "source": [
    "Load dynamic object annotations given in GeoJSON format and overlay them on the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f54778c5",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "image = visualize_annotated_objects_on_image(SEQ_FOLDER,\n",
    "                                             PROJECT_TO_PROPERTIES[DYNAMIC_OBJECTS],\n",
    "                                             project_name = DYNAMIC_OBJECTS,\n",
    "                                             images_folder = BLURRED_IMAGES)\n",
    "image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "plt.imshow(image)\n",
    "plt.title('Dynamic Objects on image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c64f1cd3",
   "metadata": {},
   "source": [
    "Load static object annotations given in GeoJSON format and overlay them on the image.\n",
    "\n",
    "Annotated objects with different classes are shown with different colors with a few of their assigned properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f8bb6ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "image = visualize_annotated_objects_on_image(SEQ_FOLDER,\n",
    "                                             PROJECT_TO_PROPERTIES[STATIC_OBJECTS],\n",
    "                                             project_name = STATIC_OBJECTS)\n",
    "image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "plt.imshow(image)\n",
    "plt.title('Static Objects on image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f42c0f26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# bird's eye view visualization of dynamic and static objects on the lidar point cloud\n",
    "bev = BEVBox()\n",
    "extracted_anno_objects = get_objects_for_bev(SEQ_FOLDER, [DYNAMIC_OBJECTS, STATIC_OBJECTS])\n",
    "image = bev(LIDAR_POINTCLOUD, extracted_anno_objects)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe65d4d4",
   "metadata": {},
   "source": [
    "Load lane marking and road painitng annotations given in GeoJSON format and overlay them on the image.\n",
    "\n",
    "The lane marking polygons are shown in red and the road painting polygons are shown in green with few of the annotated properties.\n",
    "\n",
    "Annotations contain much more detailed annotated properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "849b3a86",
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot lane markings on the image\n",
    "image = visualize_annotated_polygons_on_image(SEQ_FOLDER,\n",
    "                                              PROJECT_TO_PROPERTIES[LANE_MARKINGS],\n",
    "                                              project_name=LANE_MARKINGS)\n",
    "\n",
    "image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "plt.imshow(image)\n",
    "plt.title('Lane markings on image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bc66fc3",
   "metadata": {},
   "source": [
    "Load ego road annotations given in GeoJSON format and overlay them on the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e673883",
   "metadata": {},
   "outputs": [],
   "source": [
    "image = visualize_annotated_polygons_on_image(SEQ_FOLDER,\n",
    "                                             PROJECT_TO_PROPERTIES[EGO_ROAD],\n",
    "                                             project_name=EGO_ROAD)\n",
    "image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "plt.imshow(image)\n",
    "plt.title('Ego road on image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1f8cbe5",
   "metadata": {},
   "source": [
    "## Data sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bfd7ded",
   "metadata": {},
   "source": [
    "Zenseact Open Dataset also provides metadata information for the frames.\n",
    "\n",
    "Let's get a list of rainy frames with a decent amount of people in them.\n",
    "\n",
    "Please note that weather condition info added to the data frame is based on third-party weather info services.\n",
    "\n",
    "It can be seen from the example below that there might be some mismatched between weather info in the data frame and\n",
    "the weather seen in the visualized images. The reason is that weather info services provide information for time intervals, not real-time weather information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d538474",
   "metadata": {},
   "outputs": [],
   "source": [
    "meta_info_file = os.path.join(DATA_ROOT, \"dataframes/metadata_info.csv\")\n",
    "meta_df = pd.read_csv(meta_info_file, index_col=False)\n",
    "meta_df['time']= pd.to_datetime(meta_df['time'])\n",
    "\n",
    "with open(os.path.join(DATA_ROOT, \"dataframes/weather_codes.json\"), \"r\") as f:\n",
    "    weather_codes = json.loads(f.read())\n",
    "    weather_codes = {int(key): value for key, value in weather_codes.items()}\n",
    "\n",
    "meta_df[\"prec_decoded\"] = meta_df[\"precipitation_type\"].replace(to_replace=weather_codes).values.tolist()\n",
    "\n",
    "meta_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa048c46",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "sub_df = meta_df[meta_df.Pedestrian>50][meta_df.prec_decoded==\"rain\"]\n",
    "sub_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73e8fa26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get metadata information for a sequence\n",
    "sub_df = meta_df[meta_df.sequence_id==2545]\n",
    "sub_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c856c752",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for ind, row in sub_df.iterrows():\n",
    "\n",
    "    vehicle, camera_name, time_str = row.frame_id.split(\"_\")\n",
    "    frame_time = row.time\n",
    "    sequence_folder = \"_\".join([str(row.sequence_id), time_str])\n",
    "    OXTS_DATA = data_utils.load_oxts_from_dataset(os.path.join(OXTS_FOLDER, sequence_folder))\n",
    "    plot_gps_track_from_dataset_sequence(OXTS_DATA)\n",
    "\n",
    "    # get calibrations from dataset \n",
    "    calib = load_calib_from_json(CALIBRATIONS_FOLDER, vehicle, frame_time, camera_name)\n",
    "    \n",
    "    vision_path = os.path.join(VISION_FOLDER, sequence_folder)\n",
    "    INDEX=1\n",
    "    IMAGES, IMAGE_FILES = data_utils.load_images_from_dataset(vision_path)\n",
    "    IMAGE = IMAGES[INDEX]\n",
    "    IMAGE_FILE = IMAGE_FILES[INDEX]\n",
    "\n",
    "    # visualize GPS track over image\n",
    "    image = visualize_gps_on_image(OXTS_DATA, frame_time, calib, IMAGE)\n",
    "    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "    plt.imshow(image)\n",
    "    plt.title(f'GPS on image, sequence_id: {row.sequence_id}')\n",
    "    plt.show()\n",
    "    \n",
    "    # load lidar data for frame sequence\n",
    "    LIDAR_POINTCLOUD, LIDAR_FILES = data_utils.load_lidar_from_dataset(os.path.join(LIDAR_FOLDER, sequence_folder))\n",
    "    # visualize lidar points projections over image\n",
    "    image = visualize_lidar_on_image(LIDAR_POINTCLOUD[:, :3], calib, IMAGE)\n",
    "    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "    plt.imshow(image)\n",
    "    plt.title(f\"Lidar on image, sequence_id: {row.sequence_id}\")\n",
    "    plt.show()\n",
    "    \n",
    "    seq_folder = \"_\".join([str(row.sequence_id), time_str])\n",
    "    SEQ_FOLDER = os.path.join(DATA_ROOT, \"{}\", seq_folder)\n",
    "    for project in [DYNAMIC_OBJECTS, STATIC_OBJECTS]:\n",
    "        if row.loc[project]:\n",
    "            image = visualize_annotated_objects_on_image(SEQ_FOLDER,\n",
    "                                                         PROJECT_TO_PROPERTIES[project],\n",
    "                                                         project_name = project)\n",
    "            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "            plt.imshow(image)\n",
    "            plt.title(f\"Project: {project}, sequence_id: {row.sequence_id}\")\n",
    "            plt.show()\n",
    "    for project in [LANE_MARKINGS, EGO_ROAD]:\n",
    "        if row.loc[project]:\n",
    "            image = visualize_annotated_polygons_on_image(SEQ_FOLDER,\n",
    "                                                          PROJECT_TO_PROPERTIES[project],\n",
    "                                                          project_name = project)\n",
    "            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "            plt.imshow(image)\n",
    "            plt.title(f\"Project: {project}, sequence_id: {row.sequence_id}\")\n",
    "            plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}