How-to: Use Vagrant to Set Up a Virtual Hadoop Cluster

This guest post comes to us from David Greco, CTO of Eligotech.

Vagrant is a very nice tool for programmatically managing many virtual machines (VMs) on a single physical machine. It natively supports VirtualBox and also provides plugins for VMware Fusion and Amazon EC2, supporting the management of VMs in those environments as well.

Vagrant provides a very easy-to-use, Ruby-based internal DSL that allows the user to define one or more virtual machines together with their configuration parameters. Furthermore, it offers different mechanisms for automatic provisioning: You can use Puppet, Chef, or shell scripts for automating software installation and configuration on the machines defined in the Vagrant configuration file.

So, using Vagrant, it’s possible to define complex virtual infrastructures based on multiple VMs running on your system. Pretty cool, no?

A typical use case for Vagrant is to build working/development environments in a simple and consistent way. At my company, Eligotech, we are developing a product aimed to simplify the usage of Apache Hadoop, and CDH, Cloudera’s open source distribution, is our reference Hadoop distribution. We often need to set up a Hadoop environment on our machine for testing purposes, and we found Vagrant to be a very handy tool for that purpose.

I put together an example of a Vagrant configuration file that you can test for yourself. You’ll need to download and install Vagrant (instructions) and VirtualBox. Once everything has been installed, just copy-and-paste the text below to a file named Vagrantfile and put it in a directory named, for example, VagrantHadoop. This configuration file assumes you have at least 32GB of memory on your box; if that’s not the case, you can edit the file to suit your environment (to run fewer slaves, for example, by commenting out some of the slave configurations).

# -*- mode: ruby -*-
# vi: set ft=ruby :

$master_script = <<SCRIPT
#!/bin/bash
cat > /etc/hosts <<EOF
127.0.0.1       localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

10.211.55.100   vm-cluster-node1
10.211.55.101   vm-cluster-node2
10.211.55.102   vm-cluster-node3
10.211.55.103   vm-cluster-node4
10.211.55.104   vm-cluster-node5
10.211.55.105   vm-cluster-client
EOF

apt-get install curl -y
REPOCM=${REPOCM:-cm4}
CM_REPO_HOST=${CM_REPO_HOST:-archive.cloudera.com}
CM_MAJOR_VERSION=$(echo $REPOCM | sed -e 's/cm\\([0-9]\\).*/\\1/')
CM_VERSION=$(echo $REPOCM | sed -e 's/cm\\([0-9][0-9]*\\)/\\1/')
OS_CODENAME=$(lsb_release -sc)
OS_DISTID=$(lsb_release -si | tr '[A-Z]' '[a-z]')
if [ $CM_MAJOR_VERSION -ge 4 ]; then
  cat > /etc/apt/sources.list.d/cloudera-$REPOCM.list <<EOF
deb [arch=amd64] http://$CM_REPO_HOST/cm$CM_MAJOR_VERSION/$OS_DISTID/$OS_CODENAME/amd64/cm $OS_CODENAME-$REPOCM contrib
deb-src http://$CM_REPO_HOST/cm$CM_MAJOR_VERSION/$OS_DISTID/$OS_CODENAME/amd64/cm $OS_CODENAME-$REPOCM contrib
EOF
curl -s http://$CM_REPO_HOST/cm$CM_MAJOR_VERSION/$OS_DISTID/$OS_CODENAME/amd64/cm/archive.key > key
apt-key add key
rm key
fi
apt-get update
export DEBIAN_FRONTEND=noninteractive
apt-get -q -y --force-yes install oracle-j2sdk1.6 cloudera-manager-server-db cloudera-manager-server cloudera-manager-daemons
service cloudera-scm-server-db initdb
service cloudera-scm-server-db start
service cloudera-scm-server start
SCRIPT

$slave_script = <<SCRIPT
cat > /etc/hosts <<EOF
127.0.0.1       localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

10.211.55.100   vm-cluster-node1
10.211.55.101   vm-cluster-node2
10.211.55.102   vm-cluster-node3
10.211.55.103   vm-cluster-node4
10.211.55.104   vm-cluster-node5
10.211.55.105   vm-cluster-client
EOF
SCRIPT

$client_script = <<SCRIPT
cat > /etc/hosts <<EOF
127.0.0.1       localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

10.211.55.100   vm-cluster-node1
10.211.55.101   vm-cluster-node2
10.211.55.102   vm-cluster-node3
10.211.55.103   vm-cluster-node4
10.211.55.104   vm-cluster-node5
10.211.55.105   vm-cluster-client
EOF
SCRIPT

Vagrant.configure("2") do |config|

  config.vm.define :master do |master|
    master.vm.box = "precise64"
    master.vm.provider "vmware_fusion" do |v|
      v.vmx["memsize"]  = "4096"
    end
    master.vm.provider :virtualbox do |v|
      v.name = "vm-cluster-node1"
      v.customize ["modifyvm", :id, "--memory", "4096"]
    end
    master.vm.network :private_network, ip: "10.211.55.100"
    master.vm.hostname = "vm-cluster-node1"
    master.vm.provision :shell, :inline => $master_script
  end

  config.vm.define :slave1 do |slave1|
    slave1.vm.box = "precise64"
    slave1.vm.provider "vmware_fusion" do |v|
      v.vmx["memsize"]  = "5120"
    end
    slave1.vm.provider :virtualbox do |v|
      v.name = "vm-cluster-node2"
      v.customize ["modifyvm", :id, "--memory", "5120"]
    end
    slave1.vm.network :private_network, ip: "10.211.55.101"
    slave1.vm.hostname = "vm-cluster-node2"
    slave1.vm.provision :shell, :inline => $slave_script
  end

  config.vm.define :slave2 do |slave2|
    slave2.vm.box = "precise64"
    slave2.vm.provider "vmware_fusion" do |v|
      v.vmx["memsize"]  = "5120"
    end
    slave2.vm.provider :virtualbox do |v|
      v.name = "vm-cluster-node3"
      v.customize ["modifyvm", :id, "--memory", "5120"]
    end
    slave2.vm.network :private_network, ip: "10.211.55.102"
    slave2.vm.hostname = "vm-cluster-node3"
    slave2.vm.provision :shell, :inline => $slave_script
  end

  config.vm.define :slave3 do |slave3|
    slave3.vm.box = "precise64"
    slave3.vm.provider "vmware_fusion" do |v|
      v.vmx["memsize"]  = "5120"
    end
    slave3.vm.provider :virtualbox do |v|
      v.name = "vm-cluster-node4"
      v.customize ["modifyvm", :id, "--memory", "5120"]
    end
    slave3.vm.network :private_network, ip: "10.211.55.103"
    slave3.vm.hostname = "vm-cluster-node4"
    slave3.vm.provision :shell, :inline => $slave_script
  end

  config.vm.define :slave4 do |slave4|
    slave4.vm.box = "precise64"
    slave4.vm.provider "vmware_fusion" do |v|
      v.vmx["memsize"]  = "5120"
    end
    slave4.vm.provider :virtualbox do |v|
      v.name = "vm-cluster-node5"
      v.customize ["modifyvm", :id, "--memory", "5120"]
    end
    slave4.vm.network :private_network, ip: "10.211.55.104"
    slave4.vm.hostname = "vm-cluster-node5"
    slave4.vm.provision :shell, :inline => $slave_script
  end

  config.vm.define :client do |client|
    client.vm.box = "precise64"
    client.vm.provider "vmware_fusion" do |v|
      v.vmx["memsize"]  = "4096"
    end
    client.vm.provider :virtualbox do |v|
      v.name = "vm-cluster-client"
      v.customize ["modifyvm", :id, "--memory", "4096"]
    end
    client.vm.network :private_network, ip: "10.211.55.105"
    client.vm.hostname = "vm-cluster-client"
    client.vm.provision :shell, :inline => $client_script
  end

end

 

This file defines six machines to be assigned the following CDH roles:

  • vm-cluster-node1: This is the master; besides running the CM master, it should run the namenode, secondary namenode, and jobtracker.
  • vm-cluster-node2: This is a slave, it should run a datanode and a tasktracker.
  • vm-cluster-node3: This is a slave, it should run a datanode and a tasktracker.
  • vm-cluster-node4: This is a slave, it should run a datanode and a tasktracker.
  • vm-cluster-node5: This is a slave, it should run a datanode and a tasktracker.
  • vm-cluster-client: This machine plays the role of gateway for the cluster.

Click here to learn the meaning of the different items in the configuration file. In particular, you can see that depending on the particular provider, either VirtualBox or VMware Fusion, the memory size is changed in a different way. Observe how simple it is to switch between providers for customizing environment-specific things!

This Vagrant file does another very important thing: It installs Cloudera Manager automatically on the master node, vm-cluster-node1.

To create the virtual cluster, open a shell and just go to the directory holding the Vagrant file, i.e. VagrantHadoop. Under that directory, run:

> vagrant up --provider=virtualbox

 

After a while, depending  on how fast your machine is, Vagrant will return control — meaning that all the VMs are up and running.

At this point you are ready to configure your cluster through CM’s web UI via http://vm-cluster-node1:7180.

Have fun!

9 Responses
  • bryantrobbins / April 28, 2013 / 11:11 AM

    This is fantastic. Thank you very much for the Vagrantfile config – I’m sure I’m not the only one out there that had been trying to hack this out myself for a while.

    One thing to note is that the box the configuration depends on (assuming its Vagrant’s official “precise64″) needed to be installed manually with this command before I could vagrant up successfully:
    vagrant box add precise64 http://files.vagrantup.com/precise64.box

    Thanks for this!

  • santhikumar / May 08, 2013 / 8:48 AM

    Hi

    I could not get the http://vm-cluster-node1:7180 working .

    What’s the default username / password for these Ubuntu instances to login to each box ?

    Also the VM’s in Virtualbox are greyed out, but status shows as running.

    I’m using Windows 8 OS

    VirtualMachine version : 4.2.12
    Vagrant_1.2.2

    Any sort of help much appreciated.

    Thanks
    Santhi

  • santhikumar / May 10, 2013 / 2:51 AM

    Got the cluster setup working with Vagrant.

    Ubuntu VM login credentials are : vagrant/vagrant

    CM’s web UI login : admin/admin.

    Thanks a lot for the Vagrantfile.

  • david v / June 03, 2013 / 10:28 AM

    I can’t get the link to work on mine :( either..

    http://vm-cluster-node1:7180

    seems like the cluster is running though.

  • Tiny Tim / November 01, 2013 / 12:22 PM

    Could you put the config file up somewhere?

    I think I have some copy/pasta inspired errors:

    There is a syntax error in the following Vagrantfile. The syntax error
    message is reproduced below for convenience:

    VagrantHadoop/Vagrantfile:99: syntax error, unexpected ‘:’, expecting kEND
    master.vm.network :private_network, ip: “10.211.55.100″
    ^
    VagrantHadoop/Vagrantfile:113: syntax error, unexpected ‘:’, expecting kEND
    slave1.vm.network :private_network, ip: “10.211.55.101″

  • Tiny Tim / November 04, 2013 / 12:20 PM

    Nevermind… the version of vagrant from the repo was wicked old!

  • amit jain / November 22, 2013 / 2:28 PM

    Hi,

    Can someone do a quick video on how to build a hadoop cluster using multiple Cloudera Quick Start VMs ? so that we can really learn hadoop well in a distributed manner

    thanks
    -amit

  • cevaris / November 27, 2013 / 11:00 PM

    If you do not setup the hosts file locally, you have to ping Cloudera via with the physical address of master node.

    http://10.211.55.100:7180

  • Mike Luo / December 27, 2013 / 1:38 PM

    Thanks for the great Vagrant script.

    FWIW, I have VirtualBox 4.3.6 and have to turn Intel VT on in my BIOS to have VB works.

Leave a comment


one + = 3