Hadoop is an ideal tool to begin understanding distributed storage and distributed computing. These concepts form solid foundations for Data Engineering.
Though in production we prefer cloud-based Hadoop solutions (like AWS EMR, Cloudera CDH, etc), setting up a Hadoop cluster manually gives clarity about distributed systems.
This beginner's tutorial is written purely for educational purposes. This repository has minimal Hadoop configurations for the master and worker nodes. For more explanation, refer YouTube video https://youtu.be/c2Lg5c8v4YQ.
- step 1: Install VirtualBox on your system.
- ubuntu terminal> sudo apt install virtualbox
- step 2: Download Ubuntu Server Image.
- step 3: In virtualbox create new VM and attach the downloaded .vdi to it.
- Number of processors: 1
- RAM: 2560 GB
- Network: NAT
- step 4: Create new admin user "hduser" and login with default user.
- terminal> sudo adduser hduser
- terminal> sudo usermod -aG sudo hduser
- terminal> exit
- step 5: Login into VM with "hduser" and download & install necessary softwares.
- terminal> sudo apt update
- terminal> sudo apt install vim ssh net-tools openjdk-11-jdk git
- terminal> cd ~
- terminal> wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.2/hadoop-3.3.2.tar.gz
- terminal> tar xvf hadoop-3.3.2.tar.gz
- terminal> init 0
- step 6: In VirtualBox enable Host-only networking
- In VirtualBox -> File -> Host Network Manager -> Create
- vboxnet0 -> Enable DHCP server
- step 7: Setup VM Network
- VM -> Settings -> Network Settings
- Enable another adapter (Adapter2) as "Host-only Adapter" -> vboxnet0 -> OK
- step 8: Login into VM with "hduser" and change to static ip address.
- terminal> sudo ip a
- terminal> sudo vim /etc/netplan/01-netcfg.yaml
network: version: 2 renderer: networkd ethernets: enp0s3: dhcp4: true enp0s8: addresses: [192.168.56.50/24] dhcp4: false - terminal> sudo netplan apply
- terminal> sudo ip a
- terminal> sudo vim /etc/hosts
192.168.56.50 master 192.168.56.51 worker1 192.168.56.52 worker2 192.168.56.53 worker3 - terminal> sudo hostnamectl set-hostname master
- step 9: Configure Hadoop.
- terminal> cd ~
- terminal> vim ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 export HADOOP_HOME=$HOME/hadoop-3.3.2 export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH
- terminal> source ~/.bashrc
- terminal> git clone https://github.com/nilesh-g/hadoop-cluster-install.git
- terminal> cp hadoop-cluster-install/master/* $HADOOP_HOME/etc/hadoop/
- step 10: Examine the master configration.
- terminal> cd $HADOOP_HOME/conf
- terminal> vim hadoop-env.sh
- terminal> vim hdfs-site.xml
- terminal> vim mapred-site.xml
- terminal> vim yarn-site.xml
- terminal> vim workers
- terminal> init 0
- step 11: Create Linked Clone of this VM as "worker1". Generate new MAC addresses for all network adapters.
- step 12: Login into new VM with "hduser" and configure network and hadoop.
- terminal> sudo vim /etc/netplan/01-netcfg.yaml
network: version: 2 renderer: networkd ethernets: enp0s3: dhcp4: true enp0s8: addresses: [192.168.56.51/24] dhcp4: false - terminal> sudo netplan apply
- terminal> sudo ip a
- terminal> sudo hostnamectl set-hostname worker1
- terminal> cd ~
- terminal> cp hadoop-cluster-install/master/* $HADOOP_HOME/etc/hadoop/
- terminal> sudo vim /etc/netplan/01-netcfg.yaml
- step 13: Examine the worker configration.
- terminal> cd $HADOOP_HOME/conf
- terminal> vim hadoop-env.sh
- terminal> vim hdfs-site.xml
- terminal> vim mapred-site.xml
- terminal> vim yarn-site.xml
- terminal> init 0
- step 14: Create Linked Clone of master VM as "worker2".
- step 15: Login into new VM with hduser and configure network and hadoop.
- terminal> sudo vim /etc/netplan/01-netcfg.yaml
network: version: 2 renderer: networkd ethernets: enp0s3: dhcp4: true enp0s8: addresses: [192.168.56.52/24] dhcp4: false - terminal> sudo netplan apply
- terminal> sudo ip a
- terminal> sudo hostnamectl set-hostname worker2
- terminal> cd ~
- terminal> cp hadoop-cluster-install/master/* $HADOOP_HOME/etc/hadoop/
- terminal> init 0
- terminal> sudo vim /etc/netplan/01-netcfg.yaml
- step 16: Create Linked Clone of this VM as "worker3".
- step 17: Login into new VM with hduser and configure network and hadoop.
- terminal> sudo vim /etc/netplan/01-netcfg.yaml
network: version: 2 renderer: networkd ethernets: enp0s3: dhcp4: true enp0s8: addresses: [192.168.56.51/24] dhcp4: false - terminal> sudo netplan apply
- terminal> sudo ip a
- terminal> sudo hostnamectl set-hostname worker3
- terminal> cd ~
- terminal> cp hadoop-cluster-install/master/* $HADOOP_HOME/etc/hadoop/
- terminal> init 0
- terminal> sudo vim /etc/netplan/01-netcfg.yaml
- step 18: Start all VMs. Copy master ssh id on all workers. Ensure that master is able to login to itself and all workers without password over ssh.
- master terminal> ssh-keygen -t rsa -P ""
- master terminal> ssh-copy-id hduser@master
- master terminal> ssh-copy-id hduser@worker1
- master terminal> ssh-copy-id hduser@worker2
- master terminal> ssh-copy-id hduser@worker3
- step 19: Start Hadoop and verify.
- master terminal> hdfs namenode -format
- master terminal> start-dfs.sh
- master terminal> start-yarn.sh
- master terminal> jps
- worker1 terminal> jps
- worker2 terminal> jps
- worker3 terminal> jps
- step 20: Upload a file on HDFS to verify if cluster and replication is working properly.
- master terminal> echo "Welcome to Hadoop cluster" > hello.txt
- master terminal> hadoop fs -put /
- master terminal> hadoop fs -ls /
- master terminal> hadoop fs -cat /hello.txt
- step 21: From the host machine access Hadoop web interface.
- Browser: http://192.168.56.50:9870
- step 22: Stop the Hadoop and verify.
- master terminal> stop-yarn.sh
- master terminal> stop-dfs.sh
- step 23: Shutdown all VMs.