examples/devkit_tutorial.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "eb1c4ced",
   "metadata": {},
   "source": [
    "## Development-Kit Tutorial for Zenseact Open Dataset\n",
    "This notebook aims to introduce the ZodFrames & ZodSequences classes, which are helper classes to interact with the Frames and Sequences subsets of the Zenseact Open Dataset (ZOD) respecively. It will highlight some basic functionality that later can be used to build dataloaders in for example PyTorch.\n",
    "\n",
    "This notebook also aims to give a brief introduction to the which annotations exist and how to visualization them. \n",
    "\n",
    "#### The dataset includes data from 3 sensor modalities and calibrations for each sensor:  \n",
    "1. **Camera** - Anonymized (license plates and faces) front camera images. Available anonymization methods are:\n",
    "    - blur (Blur)\n",
    "    - dnat (Deep Fake)\n",
    "\n",
    "\n",
    "2. **LiDAR** - The LiDAR point cloud is the closest LiDAR scan to the camera timestamp of the core frame. Zenseact Open Dataset also provides a range of LiDAR point clouds captured in [-1s, +1s] at 10Hz around the core frame for the sequences.\n",
    "\n",
    "\n",
    "3. **OXTS** - High-precision GPS. OXTS data is provided in [-1s, ~10s] around the core frames for each sequence.\n",
    "\n",
    "#### There are 4 types of annotationed objects:  \n",
    "1. **dynamic_objects** - objects that can move (vehicles, pedestrians etc.) - annotated with 2D/3D bounding boxes\n",
    "2. **static_objects** - non-movable objects (light poles, traffic signs etc.) - annotated with 2D/3D bounding boxes\n",
    "3. **lane_markings** - lane markings and road paitings - annotated with polygons\n",
    "4. **ego_road** (Doesn't exist for all frames) - polygons that shows the road where ego vehicle can drive - annotated with polygons "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fa410652",
   "metadata": {},
   "source": [
    "# Initialization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51c0069f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import sys\n",
    "\n",
    "sys.path.append(\"../\")\n",
    "\n",
    "from typing import List\n",
    "\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams[\"figure.figsize\"] = [20, 10]\n",
    "\n",
    "# import the ZOD DevKit\n",
    "from zod import ZodFrames\n",
    "from zod import ZodSequences\n",
    "\n",
    "# import default constants\n",
    "import zod.constants as constants\n",
    "from zod.constants import Camera, Lidar, Anonymization, AnnotationProject\n",
    "\n",
    "# import useful data classes\n",
    "from zod.data_classes import LidarData\n",
    "\n",
    "# NOTE! Set the path to dataset and choose a version\n",
    "dataset_root = \"../data/zod\"  # your local path to zod\n",
    "version = \"mini\"  # \"mini\" or \"full\"\n",
    "\n",
    "# initialize ZodFrames\n",
    "zod_frames = ZodFrames(dataset_root=dataset_root, version=version)\n",
    "\n",
    "# initialize ZodSequences\n",
    "zod_sequences = ZodSequences(dataset_root=dataset_root, version=version)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e7f8f88",
   "metadata": {},
   "source": [
    "### Split into Training and Validation sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "525d2bca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get default training and validation splits\n",
    "training_frames = zod_frames.get_split(constants.TRAIN)\n",
    "validation_frames = zod_frames.get_split(constants.VAL)\n",
    "\n",
    "# print the number of training and validation frames\n",
    "print(f\"Number of training frames: {len(training_frames)}\")\n",
    "print(f\"Number of validation frames: {len(validation_frames)}\")\n",
    "\n",
    "training_sequences = zod_sequences.get_split(constants.TRAIN)\n",
    "validation_sequences = zod_sequences.get_split(constants.VAL)\n",
    "print(f\"Number of training sequences: {len(training_sequences)}\")\n",
    "print(f\"Number of validation sequences: {len(validation_sequences)}\")\n",
    "\n",
    "# print out the first 5 training frames\n",
    "print(\"The 5 first training frames have the ids:\", sorted(list(training_frames))[:5])\n",
    "\n",
    "# show the first training sequence\n",
    "print(\"The first training sequence has the id:\", sorted(list(training_sequences))[0])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4b1f1943",
   "metadata": {},
   "source": [
    "# ZodFrames\n",
    "### Fetch a ZodFrame\n",
    "The ZodFrames class yeild a `ZodFrame` which acts a cache for the light-weight data (e.g., ego-motion, calibration, and metadata), but also holds an `info` attribute. This in turn holds all the paths to more heavy-weight data (e.g., images and point clouds).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db74f119",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can get a specific frame by its id\n",
    "frame_from_id = zod_frames[\"009158\"]\n",
    "# or via the index\n",
    "frame_from_idx = zod_frames[9158]\n",
    "\n",
    "# these two frames are the same\n",
    "assert frame_from_id.info == frame_from_idx.info"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4eea3e11",
   "metadata": {},
   "source": [
    "### Look at some data within a ZodFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c27ac6a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "zod_frame = zod_frames[62592]\n",
    "\n",
    "# we can access the metadata of a frame\n",
    "metadata = zod_frame.metadata\n",
    "\n",
    "# print a subsample of meta data\n",
    "print(f\"Frame id: {metadata.frame_id}\")\n",
    "print(f\"Country Code: {metadata.country_code}\")\n",
    "print(f\"Time of day: {metadata.time_of_day}\")\n",
    "print(f\"Number of vehicles in the frame: {metadata.num_vehicles}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10b3f727",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can use the frame to get the OXTS of our the vehicle\n",
    "oxts = zod_frame.oxts\n",
    "print(f\"Acceleration: {oxts.accelerations.shape}\")\n",
    "print(f\"Velocities: {oxts.velocities.shape}\")\n",
    "print(f\"Poses: {oxts.poses.shape}\")\n",
    "print(f\"Timestamps: {oxts.timestamps.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "244134dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can use the frame to get the ego-motion of our the vehicle\n",
    "# note that the ego-motion is a lightwieght version of the oxts data\n",
    "ego_motion = zod_frame.ego_motion\n",
    "print(f\"Acceleration: {ego_motion.accelerations.shape}\")\n",
    "print(f\"Velocities: {ego_motion.velocities.shape}\")\n",
    "print(f\"Poses: {ego_motion.poses.shape}\")\n",
    "print(f\"Timestamps: {ego_motion.timestamps.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4497d7d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The upper-left 3x3 matrix is the rotation matrix\n",
    "rotation_matrix = zod_frame.ego_motion.poses[0, :3, :3]\n",
    "print(rotation_matrix)\n",
    "\n",
    "# The last column contain the translation\n",
    "translation = zod_frame.ego_motion.poses[0, :3, -1]\n",
    "print(translation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93fb4e4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also get the calibrations\n",
    "calibrations = zod_frame.calibration\n",
    "\n",
    "print(calibrations.lidars[Lidar.VELODYNE])\n",
    "print(calibrations.cameras[Camera.FRONT])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bcfa8a52",
   "metadata": {},
   "source": [
    "#### Camera Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f500d80",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the camera core-frame from front camera with dnat anonymization\n",
    "camera_core_frame = zod_frame.info.get_key_camera_frame(Anonymization.DNAT)\n",
    "print(camera_core_frame)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "378aecc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# one can read the image from the filepath\n",
    "image = camera_core_frame.read()\n",
    "# or use a helper directly from the frame\n",
    "image = zod_frame.get_image(Anonymization.DNAT)\n",
    "\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(image)\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d89e1407",
   "metadata": {},
   "source": [
    "#### Ego Position Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50716c57",
   "metadata": {},
   "outputs": [],
   "source": [
    "from zod.visualization.oxts_on_image import visualize_oxts_on_image\n",
    "\n",
    "zod_frame = zod_frames[\"082291\"]\n",
    "\n",
    "# extract the oxts data\n",
    "oxts = zod_frame.oxts\n",
    "\n",
    "# visualize the oxts data on the image\n",
    "calibrations = zod_frame.calibration\n",
    "\n",
    "# get the time of the keyframe (into which we want to project the oxts points)\n",
    "key_timestamp = zod_frame.info.keyframe_time.timestamp()\n",
    "\n",
    "image = zod_frame.get_image(Anonymization.DNAT)\n",
    "image = visualize_oxts_on_image(oxts, key_timestamp, calibrations, image, camera=Camera.FRONT)\n",
    "\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(image)\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "00a57dde",
   "metadata": {},
   "source": [
    "#### LiDAR Data\n",
    "Lidar fields description:\n",
    "\n",
    "| Name | Type | Units | Description |\n",
    "| --- | --- | --- | --- |\n",
    "| 'timestamp' | string |  seconds  | UTC timestamp of each point. |\n",
    "| 'x' | double |  meters  | x coordinate of the point in lidar frame |\n",
    "| 'y' | double |  meters  | y coordinate of the point in lidar frame |\n",
    "| 'z' | double |  meters  | z coordinate of the point in lidar frame |\n",
    "| 'intensity' | double |    | intensity level of each point in range [0..255] |\n",
    "| 'diode_index' | integer |    | index of diode emitter which produced a point (1..128) |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2a119c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "zod_frame = zod_frames[62592]\n",
    "\n",
    "# get the lidar core-frame\n",
    "lidar_core_frame = zod_frame.info.get_key_lidar_frame()\n",
    "print(lidar_core_frame)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f0225fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the lidar data\n",
    "pc = lidar_core_frame.read()\n",
    "\n",
    "# LidarData dataclass is a wrapper around several numpy arrays\n",
    "assert isinstance(pc, LidarData)\n",
    "\n",
    "# alternatively, we can use helper functions on the frame itself\n",
    "assert zod_frame.get_lidar()[0] == pc\n",
    "assert zod_frame.get_lidar_frames()[0].read() == pc\n",
    "\n",
    "print(f\"Points: {pc.points.shape}\")  # x, y, z\n",
    "print(f\"Timestamps: {pc.timestamps.shape}\")\n",
    "print(f\"Intensity: {pc.intensity.shape}\")\n",
    "print(f\"Diode: {pc.diode_idx.shape}\")\n",
    "\n",
    "# TODO: add visualization, e.g. 3d scatter plot with plotly"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9c27e33c",
   "metadata": {},
   "source": [
    "# Annotations\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db8920f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from zod import ObjectAnnotation\n",
    "\n",
    "# get a new frame\n",
    "zod_frame = zod_frames[\"082291\"]\n",
    "\n",
    "# get the object annotations\n",
    "annotations: List[ObjectAnnotation] = zod_frame.get_annotation(AnnotationProject.OBJECT_DETECTION)\n",
    "\n",
    "# get a single annotation object by index\n",
    "idx = 31\n",
    "print(f\"Annotation: {annotations[idx].name}\")\n",
    "\n",
    "# there are both 2d and 3d annotations\n",
    "annotation_2d = annotations[idx].box2d\n",
    "annotation_3d = annotations[idx].box3d\n",
    "print(annotation_2d)\n",
    "print(annotation_3d)\n",
    "# most objects, such as this one, have 3d annotations\n",
    "assert annotation_3d is not None\n",
    "# but some objects (small, far away, occluded) are not visible in the lidar, and thus lack 3d annotations\n",
    "assert annotations[-1].box3d is None  # like this one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3820f177",
   "metadata": {},
   "outputs": [],
   "source": [
    "from zod.visualization.object_visualization import overlay_object_2d_box_on_image, overlay_object_3d_box_on_image\n",
    "\n",
    "# we can overlay the 2d annotation on the front camera image\n",
    "camera_core_frame = zod_frame.info.get_key_camera_frame(Anonymization.DNAT)\n",
    "image = camera_core_frame.read()\n",
    "\n",
    "image = overlay_object_2d_box_on_image(image, annotation_2d, color=(255, 0, 0), line_thickness=10)\n",
    "\n",
    "plt.figure()\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(image)\n",
    "\n",
    "# we can also overlay the 3d annotation on the front camera image,\n",
    "# but for this we also need the calibrations of the sensor\n",
    "calibrations = zod_frame.calibration\n",
    "assert annotation_3d is not None\n",
    "image = overlay_object_3d_box_on_image(\n",
    "    image, annotation_3d, calibrations, color=(255, 0, 0), line_thickness=10\n",
    ")\n",
    "\n",
    "plt.figure()\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(image)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76e12472",
   "metadata": {},
   "outputs": [],
   "source": [
    "from zod import EgoRoadAnnotation\n",
    "from zod.utils.polygon_transformations import polygons_to_binary_mask\n",
    "\n",
    "zod_frame = zod_frames[9158]\n",
    "\n",
    "# get the ego road annotations and convert them to a binary mask\n",
    "annotations: List[EgoRoadAnnotation] = zod_frame.get_annotation(AnnotationProject.EGO_ROAD)\n",
    "polygons = [anno.geometry for anno in annotations]\n",
    "mask = polygons_to_binary_mask(polygons)\n",
    "\n",
    "# visualize the mask\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(mask)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25",
   "metadata": {},
   "outputs": [],
   "source": [
    "from zod import LaneAnnotation\n",
    "\n",
    "# get another frame\n",
    "zod_frame = zod_frames[23996]\n",
    "\n",
    "# get the lane markings annotations and convert to a binary mask\n",
    "annotations: List[LaneAnnotation] = zod_frame.get_annotation(constants.AnnotationProject.LANE_MARKINGS)\n",
    "polygons = [anno.geometry for anno in annotations]\n",
    "mask = polygons_to_binary_mask(polygons)\n",
    "\n",
    "# visualize the mask\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(mask)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "191b3bb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can overlay the ego road annotations on the image\n",
    "from zod.utils.polygon_transformations import polygons_to_binary_mask\n",
    "from zod.visualization.polygon_utils import overlay_mask_on_image\n",
    "\n",
    "zod_frame = zod_frames[9158]\n",
    "\n",
    "# get the camera core-frame from front camera with dnat anonymization\n",
    "camera_core_frame = zod_frame.info.get_key_camera_frame(Anonymization.DNAT)\n",
    "image = camera_core_frame.read()\n",
    "\n",
    "# get the ego road annotations and convert them to a binary mask\n",
    "ego_road_annotations = zod_frame.get_annotation(AnnotationProject.EGO_ROAD)\n",
    "ego_road_polygons = [anno.geometry for anno in ego_road_annotations]\n",
    "mask = polygons_to_binary_mask(ego_road_polygons)\n",
    "\n",
    "# overlay the mask on the image\n",
    "image = overlay_mask_on_image(mask, image, fill_color=(100, 0, 0), alpha=0.5)\n",
    "\n",
    "# visualize the mask\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(image)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c3f093e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can overlay the lane markings annotations on the image\n",
    "zod_frame = zod_frames[29229]\n",
    "\n",
    "# get the camera core-frame from front camera with dnat anonymization\n",
    "camera_core_frame = zod_frame.info.get_key_camera_frame(Anonymization.DNAT)\n",
    "\n",
    "# get the image\n",
    "image = camera_core_frame.read()\n",
    "\n",
    "# get the ego road annotations\n",
    "lane_annotations = zod_frame.get_annotation(AnnotationProject.LANE_MARKINGS)\n",
    "lane_polygons = [lane.geometry for lane in lane_annotations]\n",
    "\n",
    "# convert the polygons to a binary mask (which can be used\n",
    "# for ground truth in e.g. semantic segmentation)\n",
    "mask = polygons_to_binary_mask(lane_polygons)\n",
    "\n",
    "# overlay the mask on the image\n",
    "image = overlay_mask_on_image(mask, image, fill_color=(0, 0, 100), alpha=0.75)\n",
    "\n",
    "# visualize the mask\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(image)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da8d7764",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize LiDAR and objects in Bird's Eye View\n",
    "from zod.visualization.lidar_bev import BEVBox\n",
    "\n",
    "zod_frame = zod_frames[\"009158\"]\n",
    "\n",
    "# get the LiDAR point cloud\n",
    "pcd = zod_frame.get_lidar()[0]\n",
    "\n",
    "# get the object annotations\n",
    "object_annotations = zod_frame.get_annotation(AnnotationProject.OBJECT_DETECTION)\n",
    "\n",
    "bev = BEVBox()\n",
    "bev_image = bev(\n",
    "    np.hstack((pcd.points, pcd.intensity[:, None])),\n",
    "    (\n",
    "        np.array([obj.name for obj in object_annotations if obj.box3d]),\n",
    "        np.concatenate(\n",
    "            [obj.box3d.center[None, :] for obj in object_annotations if obj.box3d], axis=0\n",
    "        ),\n",
    "        np.concatenate(\n",
    "            [obj.box3d.size[None, :] for obj in object_annotations if obj.box3d], axis=0\n",
    "        ),\n",
    "        np.array([obj.box3d.orientation for obj in object_annotations if obj.box3d]),\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0723fb66",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also visualize the lidar point cloud in the image\n",
    "from zod.visualization.lidar_on_image import visualize_lidar_on_image\n",
    "\n",
    "zod_frame = zod_frames[\"087912\"]\n",
    "\n",
    "image = zod_frame.get_image()\n",
    "image_timestamp = zod_frame.info.keyframe_time.timestamp()\n",
    "\n",
    "# Get a single Lidar point cloud\n",
    "core_lidar = zod_frame.get_lidar()[0]\n",
    "# Motion-compensate it to the image timestamp (minorly improves alignment)\n",
    "compensated_lidar = zod_frame.compensate_lidar(core_lidar, image_timestamp)\n",
    "# Visualize by projecting the point cloud onto the image\n",
    "lid_image = visualize_lidar_on_image(\n",
    "    core_lidar,\n",
    "    zod_frame.calibration,\n",
    "    image,\n",
    ")\n",
    "plt.axis(\"off\")\n",
    "plt.title(\"Core LIDAR projected onto image\")\n",
    "plt.imshow(lid_image)\n",
    "plt.show()\n",
    "\n",
    "# Plot aggregated Lidar point cloud\n",
    "aggregated_lidar = zod_frame.get_aggregated_lidar(\n",
    "    num_before=10, num_after=0, timestamp=image_timestamp\n",
    ")\n",
    "lid_image = visualize_lidar_on_image(\n",
    "    aggregated_lidar,\n",
    "    zod_frame.calibration,\n",
    "    image,\n",
    ")\n",
    "plt.axis(\"off\")\n",
    "plt.title(\"Aggregated LIDAR projected onto image\")\n",
    "plt.imshow(lid_image)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "382a8eb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also visualize all together\n",
    "zod_frame = zod_frames[9158]\n",
    "\n",
    "pcd = zod_frame.get_aggregated_lidar(num_before=3)\n",
    "lane_annotations: List[LaneAnnotation] = zod_frame.get_annotation(AnnotationProject.LANE_MARKINGS)\n",
    "ego_road_annotations: List[EgoRoadAnnotation] = zod_frame.get_annotation(AnnotationProject.EGO_ROAD)\n",
    "object_annotations: List[ObjectAnnotation] = zod_frame.get_annotation(AnnotationProject.OBJECT_DETECTION)\n",
    "\n",
    "ego_road_mask = polygons_to_binary_mask([anno.geometry for anno in ego_road_annotations])\n",
    "lane_mask = polygons_to_binary_mask([anno.geometry for anno in lane_annotations])\n",
    "calibrations = zod_frame.calibration\n",
    "image = zod_frame.get_image(Anonymization.DNAT)\n",
    "\n",
    "# overlay the mask/annotation/pointcloud on the image\n",
    "image = visualize_lidar_on_image(pcd, calibrations, image)\n",
    "image = overlay_mask_on_image(ego_road_mask, image, fill_color=(100, 0, 0), alpha=0.5)\n",
    "image = overlay_mask_on_image(lane_mask, image, fill_color=(0, 100, 0), alpha=0.5)\n",
    "for annotation in object_annotations:\n",
    "    if annotation.box3d:\n",
    "        image = overlay_object_3d_box_on_image(\n",
    "            image, annotation.box3d, calibrations, color=(0, 100, 0), line_thickness=5\n",
    "        )\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(image)\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a94e9143",
   "metadata": {},
   "source": [
    "# ZodSequence\n",
    "Visualization functionality for ZodFrames also works on sequences. Let's take a quick look.\n",
    "\n",
    "### Fetch a ZodSequence\n",
    "The ZodSequences class yeild a `ZodSequence` which acts a cache for the light-weight data (e.g., ego-motion, calibration, and metadata), but also holds an `info` attribute. This in turn holds all the paths to more heavy-weight data (e.g., images and point clouds) for all timesteps in the sequence. Note that annotations are only provided for a single frame, namely the `key_frame`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad00d40b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can do the same for the sequences\n",
    "seq = zod_sequences[list(validation_sequences)[0]]\n",
    "\n",
    "# Get the lidar frames\n",
    "print(f\"Number of lidar frames: {len(seq.info.get_lidar_frames(lidar=Lidar.VELODYNE))}\")\n",
    "# We can also get the original camera frames\n",
    "print(f\"Number of camera frames: {len(seq.info.get_camera_frames())}\")\n",
    "\n",
    "# Or see how long the sequence is\n",
    "print(f\"Timespan: {(seq.info.end_time - seq.info.start_time).total_seconds()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86000413",
   "metadata": {},
   "outputs": [],
   "source": [
    "from zod.visualization.lidar_on_image import visualize_lidar_on_image\n",
    "\n",
    "# get the key frames\n",
    "key_camera_frame = seq.info.get_key_camera_frame()\n",
    "\n",
    "# get the annotations (when they are ready...)\n",
    "try:\n",
    "    annotations = seq.get_annotation(AnnotationProject.OBJECT_DETECTION)\n",
    "except:\n",
    "    annotations = []\n",
    "\n",
    "image = key_camera_frame.read()\n",
    "pcd = seq.get_compensated_lidar(key_camera_frame.time)\n",
    "\n",
    "image = visualize_lidar_on_image(\n",
    "    pcd,\n",
    "    seq.calibration,\n",
    "    image,\n",
    ")\n",
    "\n",
    "for annotation in annotations:\n",
    "    if annotation.box3d:\n",
    "        image = overlay_object_3d_box_on_image(\n",
    "            image, annotation.box3d, seq.calibration, color=(0, 100, 0), line_thickness=10\n",
    "        )\n",
    "\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(image)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f6dfb64",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also visualize the entire sequence\n",
    "draw_every_nth = 5  # (Only visualization every nth frame for speed...)\n",
    "\n",
    "# get the mapping between camera and lidar frames\n",
    "frames = seq.info.get_camera_lidar_map()\n",
    "\n",
    "images = []\n",
    "# iterate over the frames\n",
    "for i, frame in enumerate(frames):\n",
    "    if i % draw_every_nth == 0:\n",
    "        camera_frame, lidar_frame = frame\n",
    "\n",
    "        img = camera_frame.read()\n",
    "        pcd = seq.get_compensated_lidar(camera_frame.time)\n",
    "\n",
    "        lid_image = visualize_lidar_on_image(\n",
    "            pcd,\n",
    "            seq.calibration,\n",
    "            img,\n",
    "        )\n",
    "        images.append(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3aec82c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a gif with all the frames (requires imageio)\n",
    "import imageio\n",
    "from IPython.core.display import Image\n",
    "\n",
    "imageio.mimsave(\"test.gif\", images, duration=0.2)\n",
    "Image(filename=\"test.gif\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.9 ('zod')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "bf01abf82ecef80c32d8e931926ae3a279c50e988a79abe7f7228cbe386411b0"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}