01_What_is_Apache_Spark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Spark Image](https://upload.wikimedia.org/wikipedia/commons/thumb/f/f3/Apache_Spark_logo.svg/1200px-Apache_Spark_logo.svg.png)\n",
    "# What is Apache Spark?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apache Spark is a `unified` computing engine and a set of `libraries` for `parallel data processing` on computer clusters.\n",
    "\n",
    "On the speed side, Spark `extends` the popular `MapReduce model` to efficiently support more types of computations, including interactive queries and stream processing. One of the main features Spark offers for `speed` is the ability to run `computations in memory`, but the system is also more efficient than MapReduce for complex applications running on disk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"https://i.ibb.co/4tKqXdm/Spark.png\" alt= \"Spark\"></img>\n",
    "<br>\n",
    "<em>Simple illustration of all that Spark has to offer an end user<em></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Apache Spark’s Philosophy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Let’s break down our description of Apache Spark — a unified computing engine and set of libraries for big data — into its key components.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Unified Stack\n",
    "The Spark project contains multiple closely integrated components. \n",
    "-  At its core, Spark is a “computational engine” that is responsible for scheduling, distributing, and monitoring applications\n",
    "- It powers multiple higher-level components specialized for various workloads, such as SQL or machine learning\n",
    "- All libraries and higher level components in the stack benefit from improvements at the lower layers.\n",
    "    > Eg: when Spark’s core engine adds an optimization, SQL and machine learning libraries automatically speed up as well. \n",
    "- The costs associated with running the stack are minimized, because instead of running 5–10 independent software systems, an organization needs to run only one.\n",
    "- One of the largest advantages of tight integration is the ability to build applications that seamlessly combine different processing models.\n",
    "    > Eg: In Spark you can write one application that uses machine learning to classify data in real time as it is ingested from streaming sources. Simultaneously, analysts can query the resulting data, also in real time, via SQL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the same time that Spark strives for unification, it carefully limits its scope to a computing engine. \n",
    "- We can use Spark with a wide variety of persistent storage systems, including cloud storage systems such as Azure Storage and Amazon S3, distributed file systems such as Apache Hadoop, key-value stores such as Apache Cassandra, and message buses such as Apache Kafka. \n",
    "- Spark neither stores data long term itself, nor favors one over another.\n",
    "- Data is computationaly expensive to move so Spark focuses on performing computations over the data, no matter where it resides."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spark’s final component is its libraries, which build on its design as a unified engine to provide a unified API for common data analysis tasks. Spark’s standard libraries are actually the bulk of the open source projects\n",
    "- Spark includes libraries for SQL and structured data (Spark SQL), machine learning (MLlib), stream processing (Spark Streaming and the newer Structured Streaming), and graph analytics (GraphX). \n",
    "- Beyond these libraries, there are hundreds of open source external libraries ranging from connectors for various storage systems to machine learning algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spark’s Basic Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Single machines do not have enough power and resources to perform computations on huge amounts of information. A `cluster`, or group, of computers, pools the resources of many machines together, giving us the ability to use all the cumulative resources as if they were a single computer. ow, a group of machines alone is not powerful, you need a framework to coordinate work across them. Spark does just that, managing and coordinating the execution of\n",
    "tasks on data across a cluster of computers.\n",
    "\n",
    "The cluster of machines that Spark will use to execute tasks is managed by a cluster manager like\n",
    "`Spark’s standalone cluster manager`, `YARN`, or `Mesos`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spark Applications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Spark Applications consist of a `driver` process and a set of `executor` processes.**\n",
    "\n",
    "- The driver process is the heart of a spark application uns your main() function, sits on a node in the cluster, and is responsible for three things:\n",
    "    - maintaining information about the Spark Application\n",
    "    - responding to a user’s program or input\n",
    "    - Analyzing, distributing, and scheduling work across the executors\n",
    "- The executors are responsible for actually carrying out the work that the driver assigns them. This means that each executor is responsible for only two things:\n",
    "    - executing code assigned to it by the driver,\n",
    "    - reporting the state of the computation on that executor back to the driver node"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"https://i.ibb.co/BBKv55G/Spark-Application.png\" alt=\"Spark-Application\" border=\"0\">\n",
    "<br>\n",
    "    <em>In this illustration we see on the left, our driver and the four executors on the right. In this diagram the concept of cluster nodes is removed. The user can specify how many executors should fall on each node through configurations.</em></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **_NOTE:_**  Spark, in addition to its cluster mode, also has a local mode. The driver and executors are simply\n",
    "processes, which means that they can live on the same machine or different machines. In local mode,\n",
    "the driver and executurs run (as threads) on your individual computer instead of a cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spark’s Language APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spark’s language APIs make it possible for you to run Spark code using various programming languages.\n",
    "\n",
    "- Scala\n",
    "- Java\n",
    "- Python\n",
    "- SQL\n",
    "- R\n",
    "\n",
    "There is a `SparkSession` object available to the user, which is the entrance point to running Spark code. When using Spark from Python or R, you don’t write explicit JVM instructions; instead, you write Python and R code that Spark translates into code that it then can run on the executor JVMs.\n",
    "\n",
    "<center><img src=\"https://i.ibb.co/8B1kjnq/Language-API.png\" alt=\"Language-API\" border=\"0\">\n",
    "<br>\n",
    "<em>The relationship between the SparkSession and Spark’s Language API</em></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spark’s APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spark has two fundamental sets of APIs: the low-level `unstructured` APIs, and the higher-level `structured` APIs.<br/>\n",
    "<center>\n",
    "<img src=\"https://i.ibb.co/PW8KCG2/Spark-APIs.png\" alt=\"Spark-APIs\" border=\"0\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "*This Notebook was just an introduction to Apache Spark, We will discuss all these Libraries and APIs in the further Notebooks.*\n",
    "\n",
    "**Next UP: [Low Level APIs](./02_Low_Level_APIs.ipynb)**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}