Compute_Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Globus Compute Tutorial\n",
    "\n",
    "Globus Compute is a Function-as-a-Service (FaaS) platform that enables fire-and-forget execution of Python functions on one or more remote Globus Compute endpoints.\n",
    "\n",
    "This tutorial is configured to use a tutorial endpoint hosted by the Globus Compute team. You can setup your own endpoint on resources to which you have access by following the [Globus Compute documentation](https://globus-compute.readthedocs.io/en/latest/endpoints.html). Globus Compute endpoints can be deployed on many cloud platforms, clusters with batch schedulers (e.g., Slurm, PBS), Kubernetes, or on a local PC. After configuring an endpoint you can use it in this tutorial by simply setting the `endpoint_id` below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Globus Compute Python SDK\n",
    "\n",
    "The Globus Compute Python SDK provides programming abstractions for interacting with the Globus Compute service. Before running this tutorial you should first install the Globus Compute SDK as follows:\n",
    "\n",
    "    $ pip install globus-compute-sdk\n",
    "\n",
    "The Globus Compute SDK exposes a `Client` and `Executor` for interacting with the Globus Compute service. In order to use Globus Compute, you must first authenticate using one of hundreds of supported identity provides (e.g., your institution, ORCID, or Google).  As part of the authentication process you must grant permission for Globus Compute to access your identity information (to retrieve your email address)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import pickle\n",
    "import base64\n",
    "import globus_sdk\n",
    "from globus_sdk.scopes import AuthScopes\n",
    "\n",
    "from globus_compute_sdk import Client, Executor\n",
    "from globus_compute_sdk.serialize import CombinedCode\n",
    "from globus_compute_sdk.sdk.login_manager import AuthorizerLoginManager\n",
    "from globus_compute_sdk.sdk.login_manager.manager import ComputeScopeBuilder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note**: Here we use the public Globus Compute tutorial endpoint. You can use this endpoint to run the tutorial (the endpoint is shared with all Globus Compute users). You can also change the `endpoint_id` to the UUID of any endpoint for which you have permission to execute functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tutorial_endpoint = '4b116d3c-1703-4f8f-9f6f-39921e5864df' # Public tutorial endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an Executor to submit tasks.\n",
    "\n",
    "This will attempt to use tokens from the Jupyter Hub enviornment. An authentication flow will be used to verify your identity if tokens are not available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Try to get Globus Auth token data from the JupyterHub environment\n",
    "globus_data_raw = os.getenv(\"GLOBUS_DATA\")\n",
    "if globus_data_raw:\n",
    "    tokens = pickle.loads(base64.b64decode(os.getenv('GLOBUS_DATA')))['tokens']\n",
    "    \n",
    "    ComputeScopes = ComputeScopeBuilder()\n",
    "    \n",
    "    # Create Authorizers from the Compute and Auth tokens\n",
    "    compute_auth = globus_sdk.AccessTokenAuthorizer(tokens[ComputeScopes.resource_server]['access_token'])\n",
    "    openid_auth = globus_sdk.AccessTokenAuthorizer(tokens['auth.globus.org']['access_token'])\n",
    "    \n",
    "    # Create a Compute Client from these authorizers\n",
    "    compute_login_manager = AuthorizerLoginManager(\n",
    "        authorizers={ComputeScopes.resource_server: compute_auth,\n",
    "                     AuthScopes.resource_server: openid_auth}\n",
    "    )\n",
    "    compute_login_manager.ensure_logged_in()\n",
    "    gc = Client(login_manager=compute_login_manager, code_serialization_strategy=CombinedCode())\n",
    "    gce = Executor(endpoint_id=tutorial_endpoint, client=gc)\n",
    "else:\n",
    "    # Create an Executor and initiate an auth flow if tokens are not available\n",
    "    gc = Client(code_serialization_strategy=CombinedCode())\n",
    "    gce = Executor(endpoint_id=tutorial_endpoint, client=gc)\n",
    "print(\"Executor : \", gce)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Globus Compute 101\n",
    "\n",
    "The following example demonstrates how you can execute a function with the `Executor` interface.\n",
    "\n",
    "\n",
    "### Submitting a function\n",
    "\n",
    "To execute a function, you simply call `submit` and pass a reference to the function. \n",
    "Optionally, you may also specify any input arguments to the function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the function for remote execution\n",
    "def hello_world():\n",
    "    return \"Hello World!\"\n",
    "\n",
    "future = gce.submit(hello_world)\n",
    "\n",
    "print(\"Submit returned: \", future)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting results\n",
    "\n",
    "When you `submit` a function for execution (called a `task`), the executor will return an instance of `ComputeFuture` in lieu of the result from the function.  Futures are a common way to reference asynchronous tasks, enabling you to interrogate the future to find the status, results, exceptions, etc. without blocking to wait for results.\n",
    "\n",
    "`ComputeFuture`s returned from the `Executor` can be used in the following ways:\n",
    "* `future.done()` is a non-blocking call that returns a boolean that indicates whether the task is finished.\n",
    "* `future.result()` is a blocking call that returns the result from the task execution or raises an exception if task execution failed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Returns a boolean that indicates task completion\n",
    "future.done()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note**: It can take a few seconds to execute the first task submitted to the endpoint as it provisions resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Waits for the function to complete and returns the task result or exception on failure\n",
    "future.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Catching exceptions\n",
    "\n",
    "When a task fails and you try to get its result, the `future` will raise an exception. In the following example, the `ZeroDivisionError` exception is raised when `future.result()` is called."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def division_by_zero():\n",
    "    return 42 / 0 # This will raise a ZeroDivisionError\n",
    "\n",
    "future = gce.submit(division_by_zero)\n",
    "\n",
    "try:\n",
    "    future.result()\n",
    "except Exception as exc:\n",
    "    print(\"Globus Compute returned an exception: \", exc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions with arguments\n",
    "\n",
    "Globus Compute supports registration and execution of functions with arbitrary arguments and returned parameters. Globus Compute will serialize any `*args` and `**kwargs` when executing a function and it will serialize any return parameters or exceptions.\n",
    "\n",
    "Note: Globus Compute uses standard Python serilaization libraries (i.e., Dill). It also limits the size of input arguments and returned parameters to 10 MB. For larger input or output data we suggest using Globus.\n",
    "\n",
    "The following example shows a function that computes the sum of a list of input arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_sum(a, b):\n",
    "    return a + b\n",
    "\n",
    "future = gce.submit(get_sum, 40, 2)\n",
    "print(f\"40 + 2 = {future.result()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions with dependencies\n",
    "\n",
    "In order to execute a function on a remote endpoint, Globus Compute requires that functions explictly state all dependencies within the function body. It also requires that any dependencies (e.g., libraries, modules) are available on the endpoint on which the function will execute.  For example, in the following function, we explicitly import the datetime module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_date():\n",
    "    from datetime import date\n",
    "    return date.today()\n",
    "\n",
    "future = gce.submit(get_date)\n",
    "\n",
    "print(\"Date fetched from endpoint: \", future.result())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calling external applications\n",
    "\n",
    "While Globus Compute is designed to execute Python functions, you can easily invoke external applications that are accessible on the remote endpoint. For example, the following function calls the Linux `echo` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "def echo(name):\n",
    "    import os\n",
    "    return os.popen(\"echo Hello {} from $HOSTNAME\".format(name)).read()\n",
    "\n",
    "future = gce.submit(echo, \"World\")\n",
    "\n",
    "print(\"Echo output: \", future.result())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running functions many times\n",
    "\n",
    "One of the strengths of Globus Compute is the ease by which you can run functions many times, perhaps with different input arguments.  The following example shows how you can use the Monte Carlo method to estimate pi.\n",
    "\n",
    "Specifically, if a circle with radius $r$ is inscribed inside a square with side length $2r$, the area of the circle is $\\pi r^2$ and the area of the square is $(2r)^2$. Thus, if $N$ uniformly-distributed points are dropped at random locations within the square, approximately $N\\pi/4$ will be inside the circle and therfore we can estimate the value of $\\pi$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# function that estimates pi by placing points in a box\n",
    "def pi(num_points):\n",
    "    from random import random\n",
    "    inside = 0   \n",
    "    \n",
    "    for i in range(num_points):\n",
    "        x, y = random(), random()  # Drop a point randomly within the box.\n",
    "        if x**2 + y**2 < 1:        # Count points within the circle.\n",
    "            inside += 1  \n",
    "    return (inside*4 / num_points)\n",
    "\n",
    "\n",
    "# execute the function 3 times \n",
    "estimates = []\n",
    "for i in range(3):\n",
    "    estimates.append(gce.submit(pi, \n",
    "                               10**5))\n",
    "\n",
    "# get the results and calculate the total\n",
    "total = [future.result() for future in estimates]\n",
    "\n",
    "# print the results\n",
    "print(\"Estimates: {}\".format(total))\n",
    "print(\"Average: {:.5f}\".format(sum(total)/len(estimates)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Endpoint operations\n",
    "\n",
    "You can retrieve information about endpoints including status and information about how the endpoint is configured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gc.get_endpoint_status(tutorial_endpoint)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}