Monday, March 31, 2014

Setup KVM in Ubuntu

Also refer to Ubuntu help

Pre-installation checklist


Check that your CPU supports hardware virtualization

To run KVM, you need a processor that supports hardware virtualization. Intel and AMD both have developed extensions for their processors, deemed respectively Intel VT-x (code name Vanderpool) and AMD-V (code name Pacifica). To see if your processor supports one of these, you can review the output from this command:
egrep -c '(vmx|svm)' /proc/cpuinfo
If 0 it means that your CPU doesn't support hardware virtualization.

If 1 or more it does - but you still need to make sure that virtualization is enabled in the BIOS.
On my server :
$ egrep -c '(vmx|svm)' /proc/cpuinfo
4
$ sudo apt-get install cpu-checker
$ sudo /usr/sbin/kvm-ok
INFO: /dev/kvm does not exist
HINT:   sudo modprobe kvm_intel
INFO: Your CPU supports KVM extensions
INFO: KVM (vmx) is disabled by your BIOS
HINT: Enter your BIOS setup and enable Virtualization Technology (VT),
      and then hard poweroff/poweron your system
KVM acceleration can NOT be used

Very important :

In BIOS, you should find your Virtualization setting on the last option in the Advanced Tab, Under CPU Configuration -> "Intel Virtualization Technology". The option is disabled by default and needs to be enabled.

After enabled Virtualization Technology in BIOS setup

$ sudo /usr/sbin/kvm-ok
[sudo] password for lo:
INFO: /dev/kvm exists
KVM acceleration can be used


You can still run virtual machines, but it'll be much slower without the KVM extensions.
NOTE: You may see a message like "KVM acceleration can/can NOT be used". This is misleading and only means if KVM is *currently* available (i.e. "turned on"), *not* if it is supported.


Use a 64 bit kernel (if possible)


Running a 64 bit kernel on the host operating system is recommended but not required.
  1. To serve more than 2GB of RAM for your VMs, you must use a 64-bit kernel (see 32bit_and_64bit). On a 32-bit kernel install, you'll be limited to 2GB RAM at maximum for a given VM.
  2. Also, a 64-bit system can host both 32-bit and 64-bit guests. A 32-bit system can only host 32-bit guests.
To see if your processor is 64-bit, you can run this command:
egrep -c ' lm ' /proc/cpuinfo

If 0 is printed, it means that your CPU is not 64-bit.
If 1 or higher, it is. Note: lm stands for Long Mode which equates to a 64-bit CPU.
Now see if your running kernel is 64-bit, just issue the following command:
uname -m

x86_64 indicates a running 64-bit kernel. If you use see i386, i486, i586 or i686, you're running a 32-bit kernel.
Note: x86_64 is synonymous with amd64.

On my server :
$ egrep -c ' lm ' /proc/cpuinfo
4
$ uname -m
x86_64

Installation of KVM

Install Necessary Packages

For the following setup, we will assume that you are deploying KVM on a server, and therefore do not have any X server on the machine.
You need to install a few packages first:
Lucid (10.04) or later
$ sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
Karmic (9.10) or earlier
$ sudo aptitude install kvm libvirt-bin ubuntu-vm-builder bridge-utils
  • libvirt-bin provides libvirtd which you need to administer qemu and kvm instances using libvirt
  • qemu-kvm (kvm in Karmic and earlier) is the backend
  • ubuntu-vm-builder powerful command line tool for building virtual machines
  • bridge-utils provides a bridge from your network to the virtual machines
You might also want to install virt-viewer, for viewing instances.

Add Users to Groups

Karmic (9.10) and later
You need to ensure that your username is added to the group libvirtd:
$ sudo adduser `id -un` libvirtd
Adding user '<username>' to group 'libvirtd' ...
After this, you need to relogin so that your user becomes an effective member of the libvirtd group. The members of this group can run virtual machines. (You can also 'newgrp kvm' in a terminal, but this will affect only that terminal.)

On my server :
$ sudo adduser `id -un` libvirtd
The user <username> is already a member of `libvirtd'.

Verify Installation

You can test if your install has been successful with the following command:
$ virsh -c qemu:///system list
 Id Name                 State
----------------------------------

$
If on the other hand you get something like this:
$ virsh -c qemu:///system list
libvir: Remote error : Permission denied
error: failed to connect to the hypervisor
$
Something is wrong (e.g. you did not relogin) and you probably want to fix this before you move on. The critical point here is whether or not you have write access to /var/run/libvirt/libvirt-sock.
The sock file should have permissions similar to:
$ sudo ls -la /var/run/libvirt/libvirt-sock
srwxrwx--- 1 root libvirtd 0 2010-08-24 14:54 /var/run/libvirt/libvirt-sock
Also, /dev/kvm needs to be in the right group. If you see:
 $ ls -l /dev/kvm
crw-rw----+ 1 root root 10, 232 Jul  8 22:04 /dev/kvm
You might experience problems when creating a virtual machine. Change the device's group to kvm/libvirtd instead:
sudo chown root:libvirtd /dev/kvm

Please note that /dev/kvm will not exist, if Virtualization Technology (VT) is disabled in BIOS setup

On my server 
ls -ld /dev/kvm
crw-rw----+ 1 root kvm 10, 232 Apr  1 01:00 /dev/kvm

If needed, you can try create this file by running :
sudo mknod /dev/kvm c 10 232
sudo chown root:libvirtd /dev/kvm

Now you need to either relogin or restart the kernel modules:
rmmod kvm
modprobe -a kvm

Optional: Install virt-manager (graphical user interface)

If you are working on a desktop computer you might want to install a GUI tool to manage virtual machines.
$ sudo apt-get install virt-manager
Virtual Machine Manager will appear in Applications -> System Tools menu. First create a new connection to local QEMU instance from File -> Add Connection menu. Localhost (QEMU) should appear in the virtual machine list. Note: there already exist Localhost (QEMU Usermode) connection but this does not work at least on Ubuntu 10.04.
Create a new virtual machine by pressing the top left Create a new virtual machine toolbar button.


Check if a package is installed in Ubuntu

Here is an example to check if package vino is installed

$ dpkg -s vino
Package: vino
Status: install ok installed
Priority: optional
Section: gnome
Installed-Size: 564
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Version: 3.4.2-0ubuntu1.3
Depends: libappindicator3-1 (>= 0.2.92), libavahi-client3 (>= 0.6.16), libavahi-common3 (>= 0.6.16), libavahi-glib1 (>= 0.6.16), libc6 (>= 2.15), libcairo2 (>= 1.10.0), libdbus-glib-1-2 (>= 0.78), libgcrypt11 (>= 1.4.5), libglib2.0-0 (>= 2.31.8), libgnome-keyring0 (>= 2.20.3), libgnutls26 (>= 2.12.6.1-0), libgtk-3-0 (>= 3.0.0), libice6 (>= 1:1.0.0), libjpeg8 (>= 8c), libminiupnpc8 (>= 1.6), libnotify4 (>= 0.7.0), libsm6, libsoup2.4-1 (>= 2.24.0), libtelepathy-glib0 (>= 0.11.13), libx11-6, libxdamage1 (>= 1:1.1), libxext6, libxfixes3, libxtst6, zlib1g (>= 1:1.1.4), dconf-gsettings-backend | gsettings-backend
Recommends: gvfs
Suggests: vinagre, gnome-user-guide
Breaks: gnome-session-bin (<< 3.0)
Conffiles:
 /etc/xdg/autostart/vino-server.desktop 1fb907503747b28761bd76f597983a24
Description: VNC server for GNOME
 VNC is a protocol that allows remote display of a user's desktop. This
 package provides a VNC server that integrates with GNOME, allowing you
 to export your running desktop to another computer for remote use or
 diagnosis.
Homepage: http://live.gnome.org/Vino
Original-Maintainer: Debian GNOME Maintainers <pkg-gnome-maintainers@lists.alioth.debian.org>

$ dpkg-query -l vino
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version        Description
+++-==============-==============-============================================
ii  vino           3.4.2-0ubuntu1 VNC server for GNOME

Tuesday, March 25, 2014

Setup SSH

ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
Created directory '/home/hduser/.ssh'.
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.
The key fingerprint is:
2a:de:41:3e:67:22:c7:e5:37:d6:2c:3f:24:98:5f:99 hduser@ubuntu2
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|                 |
|      . So   o   |
|     + +o .oE    |
|    o O +.=+o    |
|   . = * o.+.    |
|    . .     ..   |
+-----------------+

ssh slave
The authenticity of host 'slave (192.168.1.196)' can't be established.
ECDSA key fingerprint is b6:3b:43:14:e4:62:72:4e:e3:e4:71:10:c5:1a:32:29.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave,192.168.1.196' (ECDSA) to the list of known hosts.
hduser@slave's password:

Welcome to Ubuntu 13.10 (GNU/Linux 3.11.0-12-generic x86_64)
exit

scp .ssh/id_rsa.pub slave:~/pub.txt
hduser@slave's password:

id_rsa.pub                                    100%  395     0.4KB/s   00:00


ssh slave
hduser@slave's password:
Welcome to Ubuntu 13.10 (GNU/Linux 3.11.0-12-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Tue Mar 25 23:58:55 EDT 2014

  System load:  0.0               Processes:             77
  Usage of /:   18.6% of 5.78GB   Users logged in:       1
  Memory usage: 3%                IP address for eth0:   192.168.1.196
  Swap usage:   0%                IP address for virbr0: 192.168.122.1

  Graph this data and manage this system at:
    https://landscape.canonical.com/

111 packages can be updated.
59 updates are security updates.

Last login: Tue Mar 25 23:55:24 2014 from 192.168.1.199


cat pub.txt >> .ssh/authorized_keys
exit

ssh slave(no password is being asked this time)




Setup static IP in Ubuntu

sudo vi /etc/network/interfaces


auto eth0  # leave this line alone
#iface eth0 inet dhcp  #comment out this line
iface eth0 inet static
address 192.168.1.199
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1

sudo service networking restart

If using Oracle virtualbox, one option to connect to the local router is to set the virtualbox network setting of the guest to Bridged Adapter.

Saturday, March 22, 2014

Tuesday, March 18, 2014

Install Hadoop on Ubuntu

Michael Noll wrote two very comprehensive tutorials about running Hadoop on Ubuntu Linux. One is for single-node cluster. The other is for multi-node cluster.

I gave it a try and installed Hadoop on one Ubuntu box.

Hadoop version : 1.2.1
Ubuntu version : 12.04

Installation

Some difference I altered :
1 The Hadoop version from download site is 1.2.1.
2 Java 6 is installed via openjdk-6-jdk

hduser@ubuntu:/var/log$ java -version
java version "1.6.0_30"
OpenJDK Runtime Environment (IcedTea6 1.13.1) (6b30-1.13.1-1ubuntu2~0.12.04.1)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

3 IPv6 is not disabled

Configuration

The following configuration files I follow the single-node document

conf/core-site.xml
conf/mapred-site.xml
conf/hdfs-site.xml

Since I used a different Java 6 package, the JAVA_HOME variable is set as the following in file conf/hadoop-env.sh

JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64

Format the HDFS filesystem

hduser@ubuntu:/usr/local/hadoop/conf$ /usr/local/hadoop/bin/hadoop namenode -format
14/03/18 00:29:26 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ubuntu/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.2.1
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG:   java = 1.6.0_30
************************************************************/
14/03/18 00:29:26 INFO util.GSet: Computing capacity for map BlocksMap
14/03/18 00:29:26 INFO util.GSet: VM type       = 64-bit
14/03/18 00:29:26 INFO util.GSet: 2.0% max memory = 932118528
14/03/18 00:29:26 INFO util.GSet: capacity      = 2^21 = 2097152 entries
14/03/18 00:29:26 INFO util.GSet: recommended=2097152, actual=2097152
14/03/18 00:29:26 INFO namenode.FSNamesystem: fsOwner=hduser
14/03/18 00:29:26 INFO namenode.FSNamesystem: supergroup=supergroup
14/03/18 00:29:26 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/03/18 00:29:26 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/03/18 00:29:26 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/03/18 00:29:26 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
14/03/18 00:29:26 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/03/18 00:29:26 INFO common.Storage: Image file /app/hadoop/tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds.
14/03/18 00:29:27 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/app/hadoop/tmp/dfs/name/current/edits
14/03/18 00:29:27 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/app/hadoop/tmp/dfs/name/current/edits
14/03/18 00:29:27 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
14/03/18 00:29:27 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu/127.0.1.1
************************************************************/

Start single-node cluster

hduser@ubuntu:~$ /usr/local/hadoop/bin/start-all.sh
starting namenode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-hduser-namenode-ubuntu.out
localhost: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-hduser-datanode-ubuntu.out
localhost: starting secondarynamenode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-hduser-secondarynamenode-ubuntu.out
starting jobtracker, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-hduser-jobtracker-ubuntu.out
localhost: starting tasktracker, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-hduser-tasktracker-ubuntu.out

Checking Hadoop processes 

hduser@ubuntu:~$ jps
17320 NameNode
17559 DataNode
17810 SecondaryNameNode
17894 JobTracker
18292 Jps
18136 TaskTracker

Stop single-node cluster

hduser@ubuntu:~$ /usr/local/hadoop/bin/stop-all.sh
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: stopping datanode
localhost: stopping secondarynamenode

hduser@ubuntu:~$ jps
21570 Jps