Basics/Gen3ButlerTutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring the Gen-3 Butler\n",
    "\n",
    "<br>Author(s): **Alex Drlica-Wagner** ([@kadrlica](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@kadrlica)), **Douglas Tucker** ([@douglasleetucker](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@douglasleetucker))\n",
    "<br>Maintainer(s): **Alex Drlica-Wagner** ([@kadrlica](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@kadrlica)), **Douglas Tucker** ([@douglasleetucker](https://github.com/LSSTScienceCollaborations/StackClub/issues/new?body=@douglasleetucker))\n",
    "<br>Level: **Intermediate**\n",
    "<br>Last Verified to Run: **2021-09-03**\n",
    "<br>Verified Stack Release: **w_2021_33**\n",
    "\n",
    "### Core Concepts\n",
    "\n",
    "This notebook provides a first look at the structure and organization of the DC2 repo created with the Gen-3 Butler. The Gen-3 Butler is still under development, so this notebook is expected to be updated after the official Gen-3 release. For the time being, be sure that you are using the verified version of the stack specified above.\n",
    "\n",
    "### Learning Objectives:\n",
    "\n",
    "This notebook lays out features of how the Gen-3 butler functions:\n",
    "\n",
    "1. Create a Gen-3 butler\n",
    "2. Programmatically explore a Gen-3 repo\n",
    "3. Get some data\n",
    "\n",
    "### Logistics\n",
    "This notebook is intended to be run at `lsst-lsp-stable.ncsa.illinois.edu` or `data.lsst.cloud` from a local git clone of the [StackClub](https://github.com/LSSTScienceCollaborations/StackClub) repo.\n",
    "\n",
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Site, host, and stack version\n",
    "! echo $EXTERNAL_INSTANCE_URL\n",
    "! echo $HOSTNAME\n",
    "! eups list -s | grep lsst_distrib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generic imports\n",
    "import os,glob\n",
    "import pylab as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stack imports\n",
    "import lsst.daf.butler  as dafButler\n",
    "import lsst.afw.display as afwDisplay"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Location of the DC2 Gen3 repository on this site\n",
    "URL = os.getenv('EXTERNAL_INSTANCE_URL')\n",
    "if URL.endswith('data.lsst.cloud'): # IDF\n",
    "    repo = \"s3://butler-us-central1-dp01\"\n",
    "elif URL.endswith('ncsa.illinois.edu'): # NCSA\n",
    "    repo = \"/repo/dc2\"\n",
    "else:\n",
    "    raise Exception(f\"Unrecognized URL: {URL}\")\n",
    "\n",
    "collection='2.2i/runs/DP0.1'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gen-3 Butler\n",
    "\n",
    "One of the strengths of the Gen-3 butler relative to Gen-2 is the ability to explore a repo and find out what it contains. Starting from scratch, we want to be able to get going *with only the path to the repo*. \n",
    "\n",
    "We can do this by creating a butler without specifying the collection (since we have no idea what collections exist at this point)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "butler = dafButler.Butler(repo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the butler created, we can now access the data `registry` (a database containing information about available data products)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "registry = butler.registry\n",
    "\n",
    "# We can examine the registry with\n",
    "#help(registry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `registry` is a good tool for investigating a repo (more on the registry schema can be found [here](https://dmtn-073.lsst.io/)). For example, we can get a list of all collections, with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in sorted(registry.queryCollections()):\n",
    "    print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is our first glimpse at the data contained in the repo, but it doesn't teach us *which* collection we are actually interested in. The names do give us some hints though...\n",
    "\n",
    "* `calib` - refers to calibration products that are used for instrument signature removal\n",
    "* `refcats` - refers to the reference catalogs\n",
    "* `skymaps` - are the geometric representations of the sky coverage\n",
    "* `u/` - collections that begin with `u/` are used for personal re-runs\n",
    "\n",
    "We can generally get access to everything we are intersted in for DC2 Run 2.2i DP0.1 by selecting the collection `2.2i/runs/DP0.1`. This is a pointer to other collections that expand out recursively... More on collections can be found here: https://pipelines.lsst.io/v/weekly/modules/lsst.daf.butler/organizing.html#collections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If this collection is a pointer to other collections, expand those out recursively.\n",
    "print(collection)\n",
    "for c in sorted(registry.queryCollections(collection,flattenChains=True)):\n",
    "    print(c, registry.getCollectionType(c))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new butler with the collection of interest\n",
    "butler = dafButler.Butler(repo,collections=collection)\n",
    "registry = butler.registry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DatasetTypes don't belong to collections, so when you query for them you always get all the DatasetTypes that belong to the repo. This is all datasetTypes that were created by anyone during any processing. There may be intermediate products that were created during processing, but no longer exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for x in sorted(registry.queryDatasetTypes()):\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to get all `DatasetRef` (which include the `dataId`) for a specific `datasetType` in a specific collection with a query like this. Note that this doesn't necessarily guarentee that the specific data set still exists on disk..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasetRefs = registry.queryDatasets(datasetType='calexp',collections=collection)\n",
    "for i,ref in enumerate(datasetRefs):\n",
    "    print(ref.dataId.full)\n",
    "    try: uri = butler.getURI(ref)\n",
    "    except: print(\"File not found...\")\n",
    "    if i > 10: break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the files are not necessarily located in a subdirectory of the repo. We can get the path from the Uniform Resource Identifier (URI) that is returned by `butler.getURI`. Note that this URI refers to a local path on the filesystem, but that does not need to be the case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Repo path: \",repo)\n",
    "uri = butler.getURI(ref)\n",
    "print(\"File URI: \",uri)\n",
    "!ls -lh {uri.ospath}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now say we want to restrict our selection to datasets associated with a specific filter. We can add that constraint to our query, but first we need to figure out what the filters are called... Looking at the dataId object, we see the attributes `physical_filter` and `band` look promising."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ref.dataId.full"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"physical_filter = {ref.dataId['physical_filter']}\")\n",
    "print(f\"band = {ref.dataId['band']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like `band` is what we want, so we put it in the `where` argument of `queryDatasets`. Let's try the $g$ band."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can also sub-select on specific properties of a data set\n",
    "datasetRefs = registry.queryDatasets(datasetType='calexp',dataId={'band': 'g'}, where='visit > 700000', collections=collection)\n",
    "for i,ref in enumerate(datasetRefs):\n",
    "    print(ref.dataId.full)\n",
    "    try: uri = butler.getURI(ref)\n",
    "    except: print(\"File not found...\")\n",
    "    if i > 10: break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now that we know what collections exist (`2.2i/runs/DP0.1` in particular), the `datasetTypes` that are defined for that collection, and the `datasetRefs` (which contain `dataIds`) for data products of the requested type. This is all the information that we need to get the dataset of interest!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the list above, let's choose one detector from one visit and grab a `calexp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We could pass the datasetRef that we found above, but since the query may \n",
    "# return results in a different order we define the dataId explicitly for reproducibility. \n",
    "dataId = {'visit': '703697', 'detector': 80, 'band':'g'}\n",
    "calexp = butler.get('calexp',dataId=dataId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to get the `src` catalog associated with this `calexp` we pass the `dataId` to the butler with the `src` datasetType. Note that this performs another query to the registry database to find the `src` catalog that matches our dataId requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We could get the src table using the dataId as we did above for the calexp, \n",
    "# but this would require the butler to perform another query of the database. \n",
    "# Instead, we can just pass the ref itself directly to butler.get\n",
    "src = butler.get('src',dataId)\n",
    "src = src.copy(True)\n",
    "src.asAstropy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now plot the `calexp` with the `src` catalog overlaid. We leave an investigation of this image as an exercise to the user :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And plot!\n",
    "afwDisplay.setDefaultBackend('matplotlib') \n",
    "fig = plt.figure(figsize=(10,8))\n",
    "afw_display = afwDisplay.Display(1)\n",
    "afw_display.scale('asinh', 'zscale')\n",
    "afw_display.mtv(calexp)\n",
    "plt.gca().axis('off')\n",
    "\n",
    "with afw_display.Buffering():\n",
    "    for s in src:\n",
    "        afw_display.dot('+', s.getX(), s.getY(), ctype=afwDisplay.RED)\n",
    "        afw_display.dot('o', s.getX(), s.getY(), size=20, ctype='orange') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case above, both the `calexp` and `src` can be found by the registry, but this will not always necessarily be the case. The `queryDimensions` method provides a more flexible way to query for multiple datasets (requiring an instance of all datasets to be available for that dataId) or ask for different dataId keys than what is used to identify the dataset (which invokes various built-in relationships). An example of this is provided below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use queryDimensions to provide more flexible access\n",
    "dataIds = registry.queryDataIds([\"visit\", \"detector\", \"band\"], datasets=[\"calexp\",\"src\"], where='visit = 703697',\n",
    "                                collections=collection)\n",
    "for i,dataId in enumerate(dataIds):\n",
    "    print(dataId.full)\n",
    "    if i > 10: break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use queryDataIds to grab the dataIds for a subset\n",
    "dataIds = registry.queryDataIds([\"visit\", \"detector\"], datasets=[\"calexp\",\"src\"], \n",
    "                                where=\"band='g' and detector=0 and visit > 700000\",\n",
    "                                collections=collection)\n",
    "for i,dataId in enumerate(dataIds):\n",
    "    print(dataId.full)\n",
    "    if i > 10: break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also get more metadata about a data product dimension. (Note that the record for an exposure and a visit are different.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for dim in ['exposure','visit','detector']:\n",
    "    print(list(registry.queryDimensionRecords(dim, where='visit = 971990 and detector=0'))[0])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some Exploration\n",
    "\n",
    "Below is a scratch space for playing with things..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "LSST",
   "language": "python",
   "name": "lsst"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}