Skip to content

Latest commit

 

History

History
119 lines (83 loc) · 2.72 KB

README.md

File metadata and controls

119 lines (83 loc) · 2.72 KB

Edge2AI Environment Setup

  • This repository contains code that will automate the install of some CDH services and detail out some training steps you can complete on your own.

Acknowledgments:

  • The bulk of the materials in this training were developed by Fabio Ghirardello and his original repository is here: IoT-Predictive Maintenance

Notes:

  • This repository was created to add some new training and automate the install of some additional tools. Specifically (DNS server & Superset).
  • This has currently only been tested to run in AWS and Azure

Install / Setup

Recommended Virtual Machines
  • This will run on a AWS (m4.2xlarge) instance with 100 GB for OS and 100 GB for CDSW volume.
  • It will also run on a AWS (m4.4xlarge) instance with 100 GB for OS and 200 GB for CDSW volume.
  • In Azure it requires a Standard_D8s_v3 or a larger Standard_D16s_v3

Pre-Reqs:

  • Build new VM with instance type (m4.2xlarge, m4.4xlarge)
  • Choose a Centos Linux 7.6+
  • SSH into your new instance
  • Make note of the 2nd volume you mounted for CDSW. ie. /dev/xvdc. This volume is mounted at xvdc in linux.

Check your mount point with lsblk

Example
lsblk

# Sample Output:
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0      2:0    1    4K  0 disk 
sda      8:0    0   30G  0 disk 
├─sda1   8:1    0  500M  0 part /boot
└─sda2   8:2    0 29.5G  0 part /
sdb      8:16   0   64G  0 disk 
└─sdb1   8:17   0   64G  0 part /mnt/resource
sdc      8:32   0  100G  0 disk 
sr0     11:0    1  628K  0 rom

Install Repository

sudo -i
yum install -y git
git clone https://github.com/tlepple/demo-edge2ai-training.git
cd ~/demo-edge2ai-training

AWS Install commands:

# sample call:
. bin/setup.sh <cloud provider: aws|azure|gcp> <cluster template> <cdsw block device>

# Here is my actual AWS code example:
. bin/setup.sh aws cdsw_template.json /dev/xvdc

Azure Install commands:

. bin/setup.sh azure cdsw_template.json /dev/sdc

Estimated install time of Components:

Service Aprox Run Time
forkedOneNode 20 Min
Superset 12 Min
DNS < 1 min
CDSW Restart 15 min
  • The following commands are only used when starting or stopping services. All services are running upon completion of initial setup...

Start / Stop services running in Cloudera Manager with these commands:

# Start all services:
cd ~/demo-edge2ai-training
. bin/start_cluster.sh

# Stop all services:
cd ~/demo-edge2ai-training
. bin/stop_cluster.sh