Openeo basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "80281fc8",
   "metadata": {},
   "source": [
    "<img src=\"https://avatars.githubusercontent.com/u/74911464?s=200&v=4\"\n",
    "     alt=\"OpenEO Platform logo\"\n",
    "     style=\"float: left; margin-right: 10px;\" />\n",
    "# OpenEO platform basics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62b5b449",
   "metadata": {},
   "source": [
    "In this notebook, we will introduce you to the most basic functionality of openEO. Please do refer to our [starters documentation](https://docs.openeo.cloud/getting-started/python/#installation) for a more elaborate introduction, or for our [detailed documentation](https://open-eo.github.io/openeo-python-client/index.html) on github."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59845559",
   "metadata": {},
   "source": [
    "We will first install and import the openeo package. Then we can make a connection to a backend. Note that you can either choose a [specific backend](https://hub.openeo.org/) you want to run on, or you can let the aggregator choose one for you depending on the functionality that you are using. We are going to choose the aggregator for now, which can be accessed through the URL *openeo.cloud*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b092aaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install openeo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4cafdcf2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9.3a2\n"
     ]
    }
   ],
   "source": [
    "import openeo\n",
    "from pathlib import Path\n",
    "\n",
    "print(openeo.client_version())\n",
    "base_path = Path(\"results/basics\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a3221872",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Authenticated using refresh token.\n"
     ]
    }
   ],
   "source": [
    "connection = openeo.connect(\"openeo.vito.be\").authenticate_oidc()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aed3ac5d",
   "metadata": {},
   "source": [
    "We can now start loading a collection. An overview of collections can be found here: https://docs.openeo.cloud/data-collections/\n",
    "\n",
    "The method load_collection is documented here: https://docs.openeo.cloud/processes/#load_collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dcb33972",
   "metadata": {},
   "outputs": [],
   "source": [
    "s2_cube = connection.load_collection(\n",
    "    \"SENTINEL2_L2A\",\n",
    "    spatial_extent={\"west\": 5.1, \"east\": 5.2, \"south\": 51.1, \"north\": 51.2},\n",
    "    temporal_extent=[\"2020-01-01\", \"2020-12-31\"],\n",
    "    bands=[\"B02\", \"B04\", \"B08\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31c6f598",
   "metadata": {},
   "source": [
    "Generally, the first step after loading in a collection will be cloud masking. There are many different ways to perform cloud masking, depending on the collection used and the criteria of your specific use case. There is some documentation on cloud masking [here](https://open-eo.github.io/openeo-python-client/cookbook/ard.html), as well as a notebook that exploits a cloud masking technique based on the SCL layer of Sentinel-2, which can be found [here](https://github.com/VITObelgium/notebook-samples/blob/master/Terrascope/Advanced/OpenEODataCubesMasking.ipynb). At Vito, we have developed our own cloud masking technique based on sen2cor, which is applied [here](https://github.com/openEOPlatform/SRR2_notebooks/blob/main/UC3%20-%20Crop%20type%20feature%20engineering%20using%20random%20forest.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25147c3a",
   "metadata": {},
   "source": [
    "There are two ways to download results in openEO. One is using a synchronous job, and one is using a batch job. Synchronous jobs are quick, but can only be used for very light jobs. Heavy jobs require the use of batch processing.\n",
    "\n",
    "We will first demonstrate downloading results using a synchronous job\n",
    "\n",
    "Note: we used max_time() so that you only need to download 1 time interval just to speed up the download process for this particular demo. To see what the method does and to see more methods of a DataCube, you can check out:\n",
    "https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.datacube.DataCube.max_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e99b02b",
   "metadata": {},
   "outputs": [],
   "source": [
    "s2_cube.max_time().download(str(base_path / \"s2_cube_syn.tif\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5aac665c",
   "metadata": {},
   "source": [
    "Downloading results using a batch job works as follows, and can be done in two different ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5daeb9f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:00:00 Job 'bb183cb7-53e4-45d7-a1e0-293a24ed1029': send 'start'\n"
     ]
    }
   ],
   "source": [
    "job = s2_cube.ndvi(nir=\"B08\", red=\"B04\").execute_batch(format=\"Gtiff\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad0c78dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-4-ae6ccdd41d20>:1: DeprecationWarning: Call to deprecated method `send_job`, use `create_job` instead.\n",
      "  job = s2_cube.ndvi(nir=\"B08\", red=\"B04\").send_job(format=\"Gtiff\")\n"
     ]
    }
   ],
   "source": [
    "job = s2_cube.ndvi(nir=\"B08\", red=\"B04\").send_job(format=\"Gtiff\")\n",
    "job.start_job()\n",
    "job.describe_job()\n",
    "# job.get_results().download_file(str(base_path / \"s2_cube_batch.tif\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33c608da",
   "metadata": {},
   "source": [
    "When our job is running we can retrieve the status, a description, or we can stop a job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f5fbd7f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running\n"
     ]
    }
   ],
   "source": [
    "print(job.status())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "92323a60",
   "metadata": {},
   "outputs": [],
   "source": [
    "job.stop_job()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c103997",
   "metadata": {},
   "source": [
    "However, generally we are interested in more than just loading and downloading data. One of the things we can do is to filter a datacube, temporally, spatially or on bands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0480abad",
   "metadata": {},
   "outputs": [],
   "source": [
    "b2_filtered_cropped = (\n",
    "    s2_cube.filter_temporal([\"2020-02-01\", \"2020-02-28\"])\n",
    "    .band(\"B02\")\n",
    "    .filter_bbox(west=5.15, east=5.16, south=51.14, north=51.16, crs=4326)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d126090c",
   "metadata": {},
   "source": [
    "When filtering on bands, we can also do band math."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c837b822",
   "metadata": {},
   "outputs": [],
   "source": [
    "B04 = s2_cube.band(\"B04\")\n",
    "B08 = s2_cube.band(\"B08\")\n",
    "ndvi_cube = (B08 - B04) / (B08 + B04)\n",
    "ndvi_cube.max_time().download(str(base_path / \"ndvi.tif\"), format=\"GTiff\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "573618dc",
   "metadata": {},
   "source": [
    "Applying mathematical operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5b2352f",
   "metadata": {},
   "outputs": [],
   "source": [
    "s2_absolute = s2_cube.apply(\"absolute\")\n",
    "s2_lambda = s2_cube.apply(lambda x: x.absolute().cos())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "3512b32f",
   "metadata": {},
   "outputs": [],
   "source": [
    "s2_red = s2_cube.reduce_dimension(dimension=\"t\", reducer=\"max\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5333c27f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openeo.processes import array_element\n",
    "\n",
    "\n",
    "def callback(data):\n",
    "    band1 = array_element(data, index=0)\n",
    "    band2 = array_element(data, index=1)\n",
    "    return band1 + 1.2 * band2\n",
    "\n",
    "\n",
    "s2_cube.reduce_dimension(dimension=\"bands\", reducer=callback)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c74f695",
   "metadata": {},
   "outputs": [],
   "source": [
    "def callback2(data):\n",
    "    return data.mean()\n",
    "\n",
    "\n",
    "s2_cube.reduce_dimension(dimension=\"t\", reducer=callback2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87e54ec7",
   "metadata": {},
   "source": [
    "Sampling features. For more information, see [here](https://open-eo.github.io/openeo-python-client/cookbook/sampling.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed2e1dbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(Path(\"resources/soy_2019.geojson\")) as fn:\n",
    "    pol = fn.readlines()[0]\n",
    "\n",
    "sampled_features = ndvi_cube.filter_spatial(json.loads(pol))\n",
    "job = sampled_features.send_job(\n",
    "    title=\"Extract points from SHub\",\n",
    "    out_format=\"netCDF\",\n",
    "    sample_by_feature=True,\n",
    ")\n",
    "job.start_job()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "c393a54e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'finished'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "job.status()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}