Spark-Docker-Example

An example of using Semantive/Spark Docker image with sbt-docker plugin to create and submit a Spark App.

Requirements

1. Docker 1.10.0+ and docker-compose 1.6.0+
2. SBT

Running

1. Create docker image of the sbt-application by executing following command inside sbt-application directory:

sbt docker

2. Run whole cluster using docker-compose (from the directory containing docker-compose.yml file):

docker-compose up

Docker-compose will run three images:

• Spark master
• Spark worker
• sbt-application (Spark job)