Install Hadoop on CentOS

[root@hadoopmaster1 ~]# yum install mysql-server
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
 * base: centos.mirror.constant.com
 * extras: mirrors.advancedhosters.com
 * updates: www.gtlib.gatech.edu
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package mysql-server.x86_64 0:5.1.73-3.el6_5 will be installed
--> Processing Dependency: mysql = 5.1.73-3.el6_5 for package: mysql-server-5.1.73-3.el6_5.x86_64
--> Processing Dependency: perl-DBI for package: mysql-server-5.1.73-3.el6_5.x86_64
--> Processing Dependency: perl-DBD-MySQL for package: mysql-server-5.1.73-3.el6_5.x86_64
--> Processing Dependency: perl(DBI) for package: mysql-server-5.1.73-3.el6_5.x86_64
--> Running transaction check
---> Package mysql.x86_64 0:5.1.73-3.el6_5 will be installed
--> Processing Dependency: mysql-libs = 5.1.73-3.el6_5 for package: mysql-5.1.73-3.el6_5.x86_64
---> Package perl-DBD-MySQL.x86_64 0:4.013-3.el6 will be installed
---> Package perl-DBI.x86_64 0:1.609-4.el6 will be installed
--> Running transaction check
---> Package mysql-libs.x86_64 0:5.1.71-1.el6 will be updated
---> Package mysql-libs.x86_64 0:5.1.73-3.el6_5 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch          Version               Repository      Size
================================================================================
Installing:
 mysql-server          x86_64        5.1.73-3.el6_5        updates        8.6 M
Installing for dependencies:
 mysql                 x86_64        5.1.73-3.el6_5        updates        894 k
 perl-DBD-MySQL        x86_64        4.013-3.el6           base           134 k
 perl-DBI              x86_64        1.609-4.el6           base           705 k
Updating for dependencies:
 mysql-libs            x86_64        5.1.73-3.el6_5        updates        1.2 M

Transaction Summary
================================================================================
Install       4 Package(s)
Upgrade       1 Package(s)

Total download size: 12 M
Is this ok [y/N]: y
Downloading Packages:
(1/5): mysql-5.1.73-3.el6_5.x86_64.rpm                   | 894 kB     00:01
(2/5): mysql-libs-5.1.73-3.el6_5.x86_64.rpm              | 1.2 MB     00:02
(3/5): mysql-server-5.1.73-3.el6_5.x86_64.rpm            | 8.6 MB     00:14
(4/5): perl-DBD-MySQL-4.013-3.el6.x86_64.rpm             | 134 kB     00:00
(5/5): perl-DBI-1.609-4.el6.x86_64.rpm                   | 705 kB     00:00
--------------------------------------------------------------------------------
Total                                           501 kB/s |  12 MB     00:23
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID c105b9de: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
Importing GPG key 0xC105B9DE:
 Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org>
 Package: centos-release-6-5.el6.centos.11.1.x86_64 (@anaconda-CentOS-201311272149.x86_64/6.5)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : mysql-libs-5.1.73-3.el6_5.x86_64                             1/6
  Installing : perl-DBI-1.609-4.el6.x86_64                                  2/6
  Installing : perl-DBD-MySQL-4.013-3.el6.x86_64                            3/6
  Installing : mysql-5.1.73-3.el6_5.x86_64                                  4/6
  Installing : mysql-server-5.1.73-3.el6_5.x86_64                           5/6
  Cleanup    : mysql-libs-5.1.71-1.el6.x86_64                               6/6
  Verifying  : perl-DBD-MySQL-4.013-3.el6.x86_64                            1/6
  Verifying  : mysql-libs-5.1.73-3.el6_5.x86_64                             2/6
  Verifying  : mysql-server-5.1.73-3.el6_5.x86_64                           3/6
  Verifying  : mysql-5.1.73-3.el6_5.x86_64                                  4/6
  Verifying  : perl-DBI-1.609-4.el6.x86_64                                  5/6
  Verifying  : mysql-libs-5.1.71-1.el6.x86_64                               6/6

Installed:
  mysql-server.x86_64 0:5.1.73-3.el6_5

Dependency Installed:
  mysql.x86_64 0:5.1.73-3.el6_5       perl-DBD-MySQL.x86_64 0:4.013-3.el6
  perl-DBI.x86_64 0:1.609-4.el6

Dependency Updated:
  mysql-libs.x86_64 0:5.1.73-3.el6_5

Complete!
[root@hadoopmaster1 ~]# /etc/init.d/mysqld start
Initializing MySQL database:  Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h hadoopmaster1 password 'new-password'

Alternatively you can run:
/usr/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd /usr/mysql-test ; perl mysql-test-run.pl

Please report any problems with the /usr/bin/mysqlbug script!

                                                           [  OK  ]
Starting mysqld:                                           [  OK  ]

[root@hadoopmaster1 ~]# mysqladmin -u root password $mysqlpassword

[root@hadoopmaster1 ~]# mysql -u root -p$mysqlpassword

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 5

Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE USER 'dbuser'@'localhost' IDENTIFIED BY 'xxxxx';

Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'localhost';

Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'dbuser'@'%' IDENTIFIED BY 'xxxxx';

Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'%';

Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;

Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'localhost' WITH GRANT OPTION;

Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'dbuser'@'%' WITH GRANT OPTION;

Query OK, 0 rows affected (0.00 sec)

mysql> exit

Bye

[root@hadoopmaster1 ~]# mysql -u dbuser -pxxxxx

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 6

Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> exit

Bye

[root@hadoopmaster1 ~]# yum install mysql-connector-java*

Loaded plugins: fastestmirror, refresh-packagekit, security

Loading mirror speeds from cached hostfile

 * base: centos.mirror.constant.com

 * extras: mirrors.advancedhosters.com

 * updates: www.gtlib.gatech.edu

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package mysql-connector-java.noarch 1:5.1.17-6.el6 will be installed

--> Processing Dependency: jta >= 1.0 for package: 1:mysql-connector-java-5.1.17-6.el6.noarch

--> Processing Dependency: slf4j for package: 1:mysql-connector-java-5.1.17-6.el6.noarch

--> Running transaction check

---> Package geronimo-specs-compat.noarch 0:1.0-3.5.M2.el6 will be installed

--> Processing Dependency: geronimo-specs = 1.0-3.5.M2.el6 for package: geronimo-specs-compat-1.0-3.5.M2.el6.noarch

---> Package slf4j.noarch 0:1.5.8-8.el6 will be installed

--> Running transaction check

---> Package geronimo-specs.noarch 0:1.0-3.5.M2.el6 will be installed

--> Processing Dependency: mx4j >= 2.0.1 for package: geronimo-specs-1.0-3.5.M2.el6.noarch

--> Processing Dependency: apache-tomcat-apis for package: geronimo-specs-1.0-3.5.M2.el6.noarch

--> Running transaction check

---> Package apache-tomcat-apis.noarch 0:0.1-1.el6 will be installed

---> Package mx4j.noarch 1:3.0.1-9.13.el6 will be installed

--> Processing Dependency: log4j >= 1.2.7 for package: 1:mx4j-3.0.1-9.13.el6.noarch

--> Processing Dependency: javamail >= 1.2-5jpp for package: 1:mx4j-3.0.1-9.13.el6.noarch

--> Processing Dependency: jakarta-commons-logging >= 1.0.1 for package: 1:mx4j-3.0.1-9.13.el6.noarch

--> Processing Dependency: bcel >= 5.0 for package: 1:mx4j-3.0.1-9.13.el6.noarch

--> Processing Dependency: axis >= 1.1 for package: 1:mx4j-3.0.1-9.13.el6.noarch

--> Processing Dependency: xml-commons-resolver for package: 1:mx4j-3.0.1-9.13.el6.noarch

--> Processing Dependency: xml-commons-apis for package: 1:mx4j-3.0.1-9.13.el6.noarch

--> Processing Dependency: xml-commons for package: 1:mx4j-3.0.1-9.13.el6.noarch

--> Processing Dependency: jaf for package: 1:mx4j-3.0.1-9.13.el6.noarch

--> Running transaction check

---> Package axis.noarch 0:1.2.1-7.3.el6_3 will be installed

--> Processing Dependency: wsdl4j for package: axis-1.2.1-7.3.el6_3.noarch

--> Processing Dependency: jaxp_parser_impl for package: axis-1.2.1-7.3.el6_3.noarch

--> Processing Dependency: jakarta-commons-httpclient for package: axis-1.2.1-7.3.el6_3.noarch

--> Processing Dependency: jakarta-commons-discovery for package: axis-1.2.1-7.3.el6_3.noarch

---> Package bcel.x86_64 0:5.2-7.2.el6 will be installed

--> Processing Dependency: regexp for package: bcel-5.2-7.2.el6.x86_64

--> Processing Dependency: java-gcj-compat for package: bcel-5.2-7.2.el6.x86_64

--> Processing Dependency: java-gcj-compat for package: bcel-5.2-7.2.el6.x86_64

--> Processing Dependency: libgcj_bc.so.1()(64bit) for package: bcel-5.2-7.2.el6.x86_64

---> Package classpathx-jaf.x86_64 0:1.0-15.4.el6 will be installed

---> Package classpathx-mail.noarch 0:1.1.1-9.4.el6 will be installed

---> Package jakarta-commons-logging.noarch 0:1.0.4-10.el6 will be installed

---> Package log4j.x86_64 0:1.2.14-6.4.el6 will be installed

---> Package xml-commons-apis.x86_64 0:1.3.04-3.6.el6 will be installed

---> Package xml-commons-resolver.x86_64 0:1.1-4.18.el6 will be installed

--> Running transaction check

---> Package jakarta-commons-discovery.noarch 1:0.4-5.4.el6 will be installed

---> Package jakarta-commons-httpclient.x86_64 1:3.1-0.7.el6_3 will be installed

---> Package java-1.5.0-gcj.x86_64 0:1.5.0.0-29.1.el6 will be installed

--> Processing Dependency: sinjdoc for package: java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64

---> Package libgcj.x86_64 0:4.4.7-4.el6 will be installed

---> Package regexp.x86_64 0:1.5-4.4.el6 will be installed

---> Package wsdl4j.noarch 0:1.5.2-7.8.el6 will be installed

---> Package xerces-j2.x86_64 0:2.7.1-12.6.el6_0 will be installed

--> Running transaction check

---> Package sinjdoc.x86_64 0:0.5-9.1.el6 will be installed

--> Processing Dependency: java_cup >= 0.10 for package: sinjdoc-0.5-9.1.el6.x86_64

--> Running transaction check

---> Package java_cup.x86_64 1:0.10k-5.el6 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

 Package                        Arch       Version               Repository

                                                                           Size

================================================================================

Installing:

 mysql-connector-java           noarch     1:5.1.17-6.el6        base     1.4 M

Installing for dependencies:

 apache-tomcat-apis             noarch     0.1-1.el6             base     164 k

 axis                           noarch     1.2.1-7.3.el6_3       base     1.5 M

 bcel                           x86_64     5.2-7.2.el6           base     1.4 M

 classpathx-jaf                 x86_64     1.0-15.4.el6          base     100 k

 classpathx-mail                noarch     1.1.1-9.4.el6         base     479 k

 geronimo-specs                 noarch     1.0-3.5.M2.el6        base     111 k

 geronimo-specs-compat          noarch     1.0-3.5.M2.el6        base     7.1 k

 jakarta-commons-discovery      noarch     1:0.4-5.4.el6         base      73 k

 jakarta-commons-httpclient     x86_64     1:3.1-0.7.el6_3       base     587 k

 jakarta-commons-logging        noarch     1.0.4-10.el6          base      52 k

 java-1.5.0-gcj                 x86_64     1.5.0.0-29.1.el6      base     139 k

 java_cup                       x86_64     1:0.10k-5.el6         base     197 k

 libgcj                         x86_64     4.4.7-4.el6           base      19 M

 log4j                          x86_64     1.2.14-6.4.el6        base     679 k

 mx4j                           noarch     1:3.0.1-9.13.el6      base     1.1 M

 regexp                         x86_64     1.5-4.4.el6           base     105 k

 sinjdoc                        x86_64     0.5-9.1.el6           base     705 k

 slf4j                          noarch     1.5.8-8.el6           base     131 k

 wsdl4j                         noarch     1.5.2-7.8.el6         base     157 k

 xerces-j2                      x86_64     2.7.1-12.6.el6_0      base     2.9 M

 xml-commons-apis               x86_64     1.3.04-3.6.el6        base     439 k

 xml-commons-resolver           x86_64     1.1-4.18.el6          base     145 k

Transaction Summary

================================================================================

Install      23 Package(s)

Total download size: 31 M

Installed size: 103 M

Is this ok [y/N]: y

Downloading Packages:

(1/23): apache-tomcat-apis-0.1-1.el6.noarch.rpm          | 164 kB     00:00

(2/23): axis-1.2.1-7.3.el6_3.noarch.rpm                  | 1.5 MB     00:00

(3/23): bcel-5.2-7.2.el6.x86_64.rpm                      | 1.4 MB     00:00

(4/23): classpathx-jaf-1.0-15.4.el6.x86_64.rpm           | 100 kB     00:00

(5/23): classpathx-mail-1.1.1-9.4.el6.noarch.rpm         | 479 kB     00:00

(6/23): geronimo-specs-1.0-3.5.M2.el6.noarch.rpm         | 111 kB     00:00

(7/23): geronimo-specs-compat-1.0-3.5.M2.el6.noarch.rpm  | 7.1 kB     00:00

(8/23): jakarta-commons-discovery-0.4-5.4.el6.noarch.rpm |  73 kB     00:00

(9/23): jakarta-commons-httpclient-3.1-0.7.el6_3.x86_64. | 587 kB     00:00

(10/23): jakarta-commons-logging-1.0.4-10.el6.noarch.rpm |  52 kB     00:00

(11/23): java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64.rpm      | 139 kB     00:00

(12/23): java_cup-0.10k-5.el6.x86_64.rpm                 | 197 kB     00:00

(13/23): libgcj-4.4.7-4.el6.x86_64.rpm                   |  19 MB     00:12

(14/23): log4j-1.2.14-6.4.el6.x86_64.rpm                 | 679 kB     00:00

(15/23): mx4j-3.0.1-9.13.el6.noarch.rpm                  | 1.1 MB     00:00

(16/23): mysql-connector-java-5.1.17-6.el6.noarch.rpm    | 1.4 MB     00:00

(17/23): regexp-1.5-4.4.el6.x86_64.rpm                   | 105 kB     00:00

(18/23): sinjdoc-0.5-9.1.el6.x86_64.rpm                  | 705 kB     00:00

(19/23): slf4j-1.5.8-8.el6.noarch.rpm                    | 131 kB     00:00

(20/23): wsdl4j-1.5.2-7.8.el6.noarch.rpm                 | 157 kB     00:00

(21/23): xerces-j2-2.7.1-12.6.el6_0.x86_64.rpm           | 2.9 MB     00:01

(22/23): xml-commons-apis-1.3.04-3.6.el6.x86_64.rpm      | 439 kB     00:00

(23/23): xml-commons-resolver-1.1-4.18.el6.x86_64.rpm    | 145 kB     00:00

--------------------------------------------------------------------------------

Total                                           1.5 MB/s |  31 MB     00:20

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

  Installing : libgcj-4.4.7-4.el6.x86_64                                   1/23

  Installing : jakarta-commons-logging-1.0.4-10.el6.noarch                 2/23

  Installing : 1:jakarta-commons-discovery-0.4-5.4.el6.noarch              3/23

  Installing : java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64                      4/23

  Installing : 1:java_cup-0.10k-5.el6.x86_64                               5/23

  Installing : sinjdoc-0.5-9.1.el6.x86_64                                  6/23

  Installing : xml-commons-apis-1.3.04-3.6.el6.x86_64                      7/23

  Installing : xerces-j2-2.7.1-12.6.el6_0.x86_64                           8/23

  Installing : xml-commons-resolver-1.1-4.18.el6.x86_64                    9/23

  Installing : classpathx-jaf-1.0-15.4.el6.x86_64                         10/23

  Installing : classpathx-mail-1.1.1-9.4.el6.noarch                       11/23

  Installing : log4j-1.2.14-6.4.el6.x86_64                                12/23

  Installing : wsdl4j-1.5.2-7.8.el6.noarch                                13/23

  Installing : regexp-1.5-4.4.el6.x86_64                                  14/23

  Installing : bcel-5.2-7.2.el6.x86_64                                    15/23

  Installing : slf4j-1.5.8-8.el6.noarch                                   16/23

  Installing : 1:jakarta-commons-httpclient-3.1-0.7.el6_3.x86_64          17/23

  Installing : axis-1.2.1-7.3.el6_3.noarch                                18/23

  Installing : 1:mx4j-3.0.1-9.13.el6.noarch                               19/23

  Installing : apache-tomcat-apis-0.1-1.el6.noarch                        20/23

  Installing : geronimo-specs-1.0-3.5.M2.el6.noarch                       21/23

  Installing : geronimo-specs-compat-1.0-3.5.M2.el6.noarch                22/23

  Installing : 1:mysql-connector-java-5.1.17-6.el6.noarch                 23/23

  Verifying  : geronimo-specs-1.0-3.5.M2.el6.noarch                        1/23

  Verifying  : classpathx-mail-1.1.1-9.4.el6.noarch                        2/23

  Verifying  : 1:jakarta-commons-discovery-0.4-5.4.el6.noarch              3/23

  Verifying  : 1:java_cup-0.10k-5.el6.x86_64                               4/23

  Verifying  : sinjdoc-0.5-9.1.el6.x86_64                                  5/23

  Verifying  : bcel-5.2-7.2.el6.x86_64                                     6/23

  Verifying  : xerces-j2-2.7.1-12.6.el6_0.x86_64                           7/23

  Verifying  : regexp-1.5-4.4.el6.x86_64                                   8/23

  Verifying  : jakarta-commons-logging-1.0.4-10.el6.noarch                 9/23

  Verifying  : wsdl4j-1.5.2-7.8.el6.noarch                                10/23

  Verifying  : log4j-1.2.14-6.4.el6.x86_64                                11/23

  Verifying  : 1:mx4j-3.0.1-9.13.el6.noarch                               12/23

  Verifying  : classpathx-jaf-1.0-15.4.el6.x86_64                         13/23

  Verifying  : axis-1.2.1-7.3.el6_3.noarch                                14/23

  Verifying  : slf4j-1.5.8-8.el6.noarch                                   15/23

  Verifying  : xml-commons-apis-1.3.04-3.6.el6.x86_64                     16/23

  Verifying  : geronimo-specs-compat-1.0-3.5.M2.el6.noarch                17/23

  Verifying  : 1:jakarta-commons-httpclient-3.1-0.7.el6_3.x86_64          18/23

  Verifying  : 1:mysql-connector-java-5.1.17-6.el6.noarch                 19/23

  Verifying  : libgcj-4.4.7-4.el6.x86_64                                  20/23

  Verifying  : apache-tomcat-apis-0.1-1.el6.noarch                        21/23

  Verifying  : xml-commons-resolver-1.1-4.18.el6.x86_64                   22/23

  Verifying  : java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64                     23/23

Installed:

  mysql-connector-java.noarch 1:5.1.17-6.el6

Dependency Installed:

  apache-tomcat-apis.noarch 0:0.1-1.el6

  axis.noarch 0:1.2.1-7.3.el6_3

  bcel.x86_64 0:5.2-7.2.el6

  classpathx-jaf.x86_64 0:1.0-15.4.el6

  classpathx-mail.noarch 0:1.1.1-9.4.el6

  geronimo-specs.noarch 0:1.0-3.5.M2.el6

  geronimo-specs-compat.noarch 0:1.0-3.5.M2.el6

  jakarta-commons-discovery.noarch 1:0.4-5.4.el6

  jakarta-commons-httpclient.x86_64 1:3.1-0.7.el6_3

  jakarta-commons-logging.noarch 0:1.0.4-10.el6

  java-1.5.0-gcj.x86_64 0:1.5.0.0-29.1.el6

  java_cup.x86_64 1:0.10k-5.el6

  libgcj.x86_64 0:4.4.7-4.el6

  log4j.x86_64 0:1.2.14-6.4.el6

  mx4j.noarch 1:3.0.1-9.13.el6

  regexp.x86_64 0:1.5-4.4.el6

  sinjdoc.x86_64 0:0.5-9.1.el6

  slf4j.noarch 0:1.5.8-8.el6

  wsdl4j.noarch 0:1.5.2-7.8.el6

  xerces-j2.x86_64 0:2.7.1-12.6.el6_0

  xml-commons-apis.x86_64 0:1.3.04-3.6.el6

  xml-commons-resolver.x86_64 0:1.1-4.18.el6

Complete!

[root@hadoopmaster1 ~]# yum install java-1.7.0-openjdk java-1.7.0-openjdk-devel

Loaded plugins: fastestmirror, refresh-packagekit, security

Loading mirror speeds from cached hostfile

 * base: centos.mirror.constant.com

 * extras: mirrors.advancedhosters.com

 * updates: www.gtlib.gatech.edu

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package java-1.7.0-openjdk.x86_64 1:1.7.0.45-2.4.3.3.el6 will be updated

---> Package java-1.7.0-openjdk.x86_64 1:1.7.0.65-2.5.1.2.el6_5 will be an update

---> Package java-1.7.0-openjdk-devel.x86_64 1:1.7.0.65-2.5.1.2.el6_5 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

 Package                    Arch     Version                    Repository

                                                                           Size

================================================================================

Installing:

 java-1.7.0-openjdk-devel   x86_64   1:1.7.0.65-2.5.1.2.el6_5   updates   9.4 M

Updating:

 java-1.7.0-openjdk         x86_64   1:1.7.0.65-2.5.1.2.el6_5   updates    26 M

Transaction Summary

================================================================================

Install       1 Package(s)

Upgrade       1 Package(s)

Total download size: 35 M

Is this ok [y/N]: y

Downloading Packages:

(1/2): java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64. |  26 MB     00:37

(2/2): java-1.7.0-openjdk-devel-1.7.0.65-2.5.1.2.el6_5.x | 9.4 MB     00:13

--------------------------------------------------------------------------------

Total                                           704 kB/s |  35 MB     00:51

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

  Updating   : 1:java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64           1/3

  Installing : 1:java-1.7.0-openjdk-devel-1.7.0.65-2.5.1.2.el6_5.x86_64     2/3

  Cleanup    : 1:java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64             3/3

  Verifying  : 1:java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64           1/3

  Verifying  : 1:java-1.7.0-openjdk-devel-1.7.0.65-2.5.1.2.el6_5.x86_64     2/3

  Verifying  : 1:java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64             3/3

Installed:

  java-1.7.0-openjdk-devel.x86_64 1:1.7.0.65-2.5.1.2.el6_5

Updated:

  java-1.7.0-openjdk.x86_64 1:1.7.0.65-2.5.1.2.el6_5

Complete!

mkdir /usr/java
ln -s /usr/lib/jvm/java-1.7.0-openjdk.x86_64 /usr/java/default
mv /usr/bin/java /usr/bin/java.old
ln -s /usr/java/default/bin/java /usr/bin/java

Set up your environment to define JAVA_HOME to put the Java Virtual Machine and the Java compiler on your path
export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH

[root@hadoopmaster1 ~]# java -version

java version "1.7.0_65"

OpenJDK Runtime Environment (rhel-2.5.1.2.el6_5-x86_64 u65-b17)

OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

[root@hadoopmaster1 ~]# wget -nv http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.1.5.0/hdp.repo -O /etc/yum.repos.d/hdp.repo
2014-09-03 21:46:35 URL:http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.1.5.0/hdp.repo [590/590] -> "/etc/yum.repos.d/hdp.repo" [1]
[root@hadoopmaster1 ~]# yum repolist
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
 * base: centos.mirror.constant.com
 * extras: mirrors.advancedhosters.com
 * updates: www.gtlib.gatech.edu
HDP-2.1.5.0                                              | 2.9 kB     00:00
HDP-2.1.5.0/primary_db                                   |  52 kB     00:00
HDP-UTILS-1.1.0.19                                       | 2.9 kB     00:00
HDP-UTILS-1.1.0.19/primary_db                            |  31 kB     00:00
repo id            repo name                                              status
HDP-2.1.5.0        Hortonworks Data Platform Version - HDP-2.1.5.0           98
HDP-UTILS-1.1.0.19 Hortonworks Data Platform Utils Version - HDP-UTILS-1.    48
base               CentOS-6 - Base                                        6,367
extras             CentOS-6 - Extras                                         15
updates            CentOS-6 - Updates                                     1,462
repolist: 7,990

[root@hadoopmaster1 ~]# setenforce 0

[root@hadoopmaster1 ~]# getenforce

Permissive

[root@hadoopmaster1 ~]# vi /etc/sysconfig/selinux

SELINUX=permissive

[root@hadoopmaster1 ~]# chkconfig iptables off

[root@hadoopmaster1 ~]# /etc/init.d/iptables stop

iptables: Setting chains to policy ACCEPT: filter          [  OK  ]

iptables: Flushing firewall rules:                         [  OK  ]

iptables: Unloading modules:                               [  OK  ]

cd /home/hadoop/Downloads

[root@hadoopmaster1 Downloads]# wget http://public-repo-1.hortonworks.com/HDP/tools/2.1.5.0/hdp_manual_install_rpm_helper_files-2.1.5.695.tar.gz

--2014-09-03 22:20:23--  http://public-repo-1.hortonworks.com/HDP/tools/2.1.5.0/hdp_manual_install_rpm_helper_files-2.1.5.695.tar.gz

Resolving public-repo-1.hortonworks.com... 204.246.169.122, 54.230.37.202, 54.230.39.45, ...

Connecting to public-repo-1.hortonworks.com|204.246.169.122|:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 87355 (85K) [application/x-tar]

Saving to: “hdp_manual_install_rpm_helper_files-2.1.5.695.tar.gz”

100%[======================================>] 87,355      --.-K/s   in 0.07s

2014-09-03 22:20:23 (1.26 MB/s) - “hdp_manual_install_rpm_helper_files-2.1.5.695.tar.gz” saved [87355/87355]

# gunzip hdp_manual_install_rpm_helper_files-2.1.5.695.tar.gz

# tar xvf hdp_manual_install_rpm_helper_files-2.1.5.695.tar

cd /home/hadoop/Downloads/hdp_manual_install_rpm_helper_files-2.1.5.695/scripts

# cp directories.sh directories.sh.bak

# vi directories.sh

DFS_NAME_DIR="/grid/hadoop/hdfs/nn";

DFS_DATA_DIR="/grid/hadoop/hdfs/dn";

FS_CHECKPOINT_DIR="/grid/hadoop/hdfs/snn";

YARN_LOCAL_DIR="/grid/hadoop/yarn/local";

YARN_LOCAL_LOG_DIR="/grid/hadoop/yarn/logs";

ZOOKEEPER_DATA_DIR="/grid/hadoop/zookeeper/data";

[root@hadoopmaster1 ~]# umask 0022

[root@hadoopmaster1 ~]# yum install hadoop hadoop-hdfs hadoop-libhdfs hadoop-yarn hadoop-mapreduce hadoop-client openssl

Loaded plugins: fastestmirror, refresh-packagekit, security

Loading mirror speeds from cached hostfile

 * base: centos.mirror.constant.com

 * extras: mirrors.advancedhosters.com

 * updates: www.gtlib.gatech.edu

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package hadoop.x86_64 0:2.4.0.2.1.5.0-695.el6 will be installed

--> Processing Dependency: zookeeper >= 3.4.0 for package: hadoop-2.4.0.2.1.5.0-695.el6.x86_64

--> Processing Dependency: nc for package: hadoop-2.4.0.2.1.5.0-695.el6.x86_64

---> Package hadoop-client.x86_64 0:2.4.0.2.1.5.0-695.el6 will be installed

---> Package hadoop-hdfs.x86_64 0:2.4.0.2.1.5.0-695.el6 will be installed

--> Processing Dependency: bigtop-jsvc for package: hadoop-hdfs-2.4.0.2.1.5.0-695.el6.x86_64

---> Package hadoop-libhdfs.x86_64 0:2.4.0.2.1.5.0-695.el6 will be installed

---> Package hadoop-mapreduce.x86_64 0:2.4.0.2.1.5.0-695.el6 will be installed

---> Package hadoop-yarn.x86_64 0:2.4.0.2.1.5.0-695.el6 will be installed

---> Package openssl.x86_64 0:1.0.1e-15.el6 will be updated

---> Package openssl.x86_64 0:1.0.1e-16.el6_5.15 will be an update

--> Running transaction check

---> Package bigtop-jsvc.x86_64 0:1.0.10-1.el6 will be installed

---> Package nc.x86_64 0:1.84-22.el6 will be installed

---> Package zookeeper.noarch 0:3.4.5.2.1.5.0-695.el6 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

 Package             Arch      Version                     Repository      Size

================================================================================

Installing:

 hadoop              x86_64    2.4.0.2.1.5.0-695.el6       HDP-2.1.5.0     20 M

 hadoop-client       x86_64    2.4.0.2.1.5.0-695.el6       HDP-2.1.5.0     16 k

 hadoop-hdfs         x86_64    2.4.0.2.1.5.0-695.el6       HDP-2.1.5.0     15 M

 hadoop-libhdfs      x86_64    2.4.0.2.1.5.0-695.el6       HDP-2.1.5.0     34 k

 hadoop-mapreduce    x86_64    2.4.0.2.1.5.0-695.el6       HDP-2.1.5.0     26 M

 hadoop-yarn         x86_64    2.4.0.2.1.5.0-695.el6       HDP-2.1.5.0     14 M

Updating:

 openssl             x86_64    1.0.1e-16.el6_5.15          updates        1.5 M

Installing for dependencies:

 bigtop-jsvc         x86_64    1.0.10-1.el6                HDP-2.1.5.0     27 k

 nc                  x86_64    1.84-22.el6                 base            57 k

 zookeeper           noarch    3.4.5.2.1.5.0-695.el6       HDP-2.1.5.0    6.2 M

Transaction Summary

================================================================================

Install       9 Package(s)

Upgrade       1 Package(s)

Total download size: 83 M

Is this ok [y/N]: y

Downloading Packages:

(1/10): bigtop-jsvc-1.0.10-1.el6.x86_64.rpm              |  27 kB     00:00

(2/10): hadoop-2.4.0.2.1.5.0-695.el6.x86_64.rpm          |  20 MB     00:12

(3/10): hadoop-client-2.4.0.2.1.5.0-695.el6.x86_64.rpm   |  16 kB     00:00

(4/10): hadoop-hdfs-2.4.0.2.1.5.0-695.el6.x86_64.rpm     |  15 MB     00:08

(5/10): hadoop-libhdfs-2.4.0.2.1.5.0-695.el6.x86_64.rpm  |  34 kB     00:00

(6/10): hadoop-mapreduce-2.4.0.2.1.5.0-695.el6.x86_64.rp |  26 MB     00:15

(7/10): hadoop-yarn-2.4.0.2.1.5.0-695.el6.x86_64.rpm     |  14 MB     00:07

(8/10): nc-1.84-22.el6.x86_64.rpm                        |  57 kB     00:00

(9/10): openssl-1.0.1e-16.el6_5.15.x86_64.rpm            | 1.5 MB     00:02

(10/10): zookeeper-3.4.5.2.1.5.0-695.el6.noarch.rpm      | 6.2 MB     00:03

--------------------------------------------------------------------------------

Total                                           1.3 MB/s |  83 MB     01:01

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

  Installing : nc-1.84-22.el6.x86_64                                       1/11

  Installing : zookeeper-3.4.5.2.1.5.0-695.el6.noarch                      2/11

groupadd: group 'hadoop' already exists

  Installing : hadoop-2.4.0.2.1.5.0-695.el6.x86_64                         3/11

  Installing : hadoop-yarn-2.4.0.2.1.5.0-695.el6.x86_64                    4/11

  Installing : hadoop-mapreduce-2.4.0.2.1.5.0-695.el6.x86_64               5/11

  Installing : bigtop-jsvc-1.0.10-1.el6.x86_64                             6/11

  Installing : hadoop-hdfs-2.4.0.2.1.5.0-695.el6.x86_64                    7/11

  Installing : hadoop-libhdfs-2.4.0.2.1.5.0-695.el6.x86_64                 8/11

  Installing : hadoop-client-2.4.0.2.1.5.0-695.el6.x86_64                  9/11

  Updating   : openssl-1.0.1e-16.el6_5.15.x86_64                          10/11

  Cleanup    : openssl-1.0.1e-15.el6.x86_64                               11/11

  Verifying  : openssl-1.0.1e-16.el6_5.15.x86_64                           1/11

  Verifying  : hadoop-hdfs-2.4.0.2.1.5.0-695.el6.x86_64                    2/11

  Verifying  : bigtop-jsvc-1.0.10-1.el6.x86_64                             3/11

  Verifying  : zookeeper-3.4.5.2.1.5.0-695.el6.noarch                      4/11

  Verifying  : hadoop-libhdfs-2.4.0.2.1.5.0-695.el6.x86_64                 5/11

  Verifying  : hadoop-mapreduce-2.4.0.2.1.5.0-695.el6.x86_64               6/11

  Verifying  : hadoop-yarn-2.4.0.2.1.5.0-695.el6.x86_64                    7/11

  Verifying  : nc-1.84-22.el6.x86_64                                       8/11

  Verifying  : hadoop-client-2.4.0.2.1.5.0-695.el6.x86_64                  9/11

  Verifying  : hadoop-2.4.0.2.1.5.0-695.el6.x86_64                        10/11

  Verifying  : openssl-1.0.1e-15.el6.x86_64                               11/11

Installed:

  hadoop.x86_64 0:2.4.0.2.1.5.0-695.el6

  hadoop-client.x86_64 0:2.4.0.2.1.5.0-695.el6

  hadoop-hdfs.x86_64 0:2.4.0.2.1.5.0-695.el6

  hadoop-libhdfs.x86_64 0:2.4.0.2.1.5.0-695.el6

  hadoop-mapreduce.x86_64 0:2.4.0.2.1.5.0-695.el6

  hadoop-yarn.x86_64 0:2.4.0.2.1.5.0-695.el6

Dependency Installed:

  bigtop-jsvc.x86_64 0:1.0.10-1.el6               nc.x86_64 0:1.84-22.el6

  zookeeper.noarch 0:3.4.5.2.1.5.0-695.el6

Updated:

  openssl.x86_64 0:1.0.1e-16.el6_5.15

Complete!

[root@hadoopmaster1 ~]# yum install snappy snappy-devel

Loaded plugins: fastestmirror, refresh-packagekit, security

Loading mirror speeds from cached hostfile

 * base: centos.mirror.constant.com

 * extras: mirrors.advancedhosters.com

 * updates: www.gtlib.gatech.edu

Setting up Install Process

Package snappy-1.1.0-1.el6.x86_64 already installed and latest version

Resolving Dependencies

--> Running transaction check

---> Package snappy-devel.x86_64 0:1.1.0-1.el6 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

 Package               Arch            Version              Repository     Size

================================================================================

Installing:

 snappy-devel          x86_64          1.1.0-1.el6          base           14 k

Transaction Summary

================================================================================

Install       1 Package(s)

Total download size: 14 k

Installed size: 29 k

Is this ok [y/N]: y

Downloading Packages:

snappy-devel-1.1.0-1.el6.x86_64.rpm                      |  14 kB     00:00

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

  Installing : snappy-devel-1.1.0-1.el6.x86_64                              1/1

  Verifying  : snappy-devel-1.1.0-1.el6.x86_64                              1/1

Installed:

  snappy-devel.x86_64 0:1.1.0-1.el6

Complete!

[root@hadoopmaster1 ~]# ln -sf /usr/lib64/libsnappy.so /usr/lib/hadoop/lib/native/.

[root@hadoopmaster1 ~]# ls -al /usr/lib/hadoop/lib/native/libsnappy.so

lrwxrwxrwx. 1 root root 23 Sep  3 23:19 /usr/lib/hadoop/lib/native/libsnappy.so -> /usr/lib64/libsnappy.so

[root@hadoopmaster1 ~]# yum install lzo lzo-devel hadoop-lzo hadoop-lzo-native

Loaded plugins: fastestmirror, refresh-packagekit, security

Loading mirror speeds from cached hostfile

 * base: centos.mirror.constant.com

 * extras: mirrors.advancedhosters.com

 * updates: www.gtlib.gatech.edu

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package hadoop-lzo.x86_64 0:0.6.0-1 will be installed

---> Package hadoop-lzo-native.x86_64 0:0.6.0-1 will be installed

---> Package lzo.x86_64 0:2.03-3.1.el6 will be updated

---> Package lzo.x86_64 0:2.06-1.el6 will be an update

---> Package lzo-devel.x86_64 0:2.06-1.el6 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

 Package               Arch       Version          Repository              Size

================================================================================

Installing:

 hadoop-lzo            x86_64     0.6.0-1          HDP-UTILS-1.1.0.19      59 k

 hadoop-lzo-native     x86_64     0.6.0-1          HDP-UTILS-1.1.0.19      22 k

 lzo-devel             x86_64     2.06-1.el6       HDP-UTILS-1.1.0.19      31 k

Updating:

 lzo                   x86_64     2.06-1.el6       HDP-UTILS-1.1.0.19      55 k

Transaction Summary

================================================================================

Install       3 Package(s)

Upgrade       1 Package(s)

Total download size: 167 k

Is this ok [y/N]: y

Downloading Packages:

(1/4): hadoop-lzo-0.6.0-1.x86_64.rpm                     |  59 kB     00:00

(2/4): hadoop-lzo-native-0.6.0-1.x86_64.rpm              |  22 kB     00:00

(3/4): lzo-2.06-1.el6.x86_64.rpm                         |  55 kB     00:00

(4/4): lzo-devel-2.06-1.el6.x86_64.rpm                   |  31 kB     00:00

--------------------------------------------------------------------------------

Total                                           262 kB/s | 167 kB     00:00

warning: rpmts_HdrFromFdno: Header V4 RSA/SHA1 Signature, key ID 07513cad: NOKEY

Retrieving key from http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.1.5.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

Importing GPG key 0x07513CAD:

 Userid: "Jenkins (HDP Builds) <jenkin@hortonworks.com>"

 From  : http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.1.5.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

Is this ok [y/N]: y

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

  Updating   : lzo-2.06-1.el6.x86_64                                        1/5

  Installing : hadoop-lzo-0.6.0-1.x86_64                                    2/5

  Installing : lzo-devel-2.06-1.el6.x86_64                                  3/5

  Installing : hadoop-lzo-native-0.6.0-1.x86_64                             4/5

  Cleanup    : lzo-2.03-3.1.el6.x86_64                                      5/5

  Verifying  : hadoop-lzo-native-0.6.0-1.x86_64                             1/5

  Verifying  : hadoop-lzo-0.6.0-1.x86_64                                    2/5

  Verifying  : lzo-devel-2.06-1.el6.x86_64                                  3/5

  Verifying  : lzo-2.06-1.el6.x86_64                                        4/5

  Verifying  : lzo-2.03-3.1.el6.x86_64                                      5/5

Installed:

  hadoop-lzo.x86_64 0:0.6.0-1          hadoop-lzo-native.x86_64 0:0.6.0-1

  lzo-devel.x86_64 0:2.06-1.el6

Updated:

  lzo.x86_64 0:2.06-1.el6

Complete!

cp /home/hadoop/Downloads/hdp_manual_install_rpm_helper_files-2.1.5.695/scripts/directories.sh directories_profile

cp /home/hadoop/Downloads/hdp_manual_install_rpm_helper_files-2.1.5.695/scripts/usersAndGroups.sh usersAndGroups_profile

[root@hadoopmaster1 hadoop]# . directories_profile

[root@hadoopmaster1 hadoop]# . usersAndGroups_profile

[root@hadoopmaster1 hadoop]# echo $DFS_NAME_DIR

/grid/hadoop/hdfs/nn

[root@hadoopmaster1 hadoop]# mkdir -p $DFS_NAME_DIR;

[root@hadoopmaster1 hadoop]# ls -ld $DFS_NAME_DIR

drwxr-xr-x. 2 root root 4096 Sep  3 23:37 /grid/hadoop/hdfs/nn

[root@hadoopmaster1 hadoop]# chown -R $HDFS_USER:$HADOOP_GROUP $DFS_NAME_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 755 $DFS_NAME_DIR;

[root@hadoopmaster1 hadoop]# ls -ld $DFS_NAME_DIR

drwxr-xr-x. 2 hdfs hadoop 4096 Sep  3 23:37 /grid/hadoop/hdfs/nn

[root@hadoopmaster1 hadoop]# mkdir -p $FS_CHECKPOINT_DIR;

[root@hadoopmaster1 hadoop]# chown -R $HDFS_USER:$HADOOP_GROUP $FS_CHECKPOINT_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 755 $FS_CHECKPOINT_DIR;

[root@hadoopmaster1 hadoop]# ls -ld $FS_CHECKPOINT_DIR

drwxr-xr-x. 2 hdfs hadoop 4096 Sep  3 23:39 /grid/hadoop/hdfs/snn

[root@hadoopmaster1 hadoop]# mkdir -p $DFS_DATA_DIR;

[root@hadoopmaster1 hadoop]# chown -R $HDFS_USER:$HADOOP_GROUP $DFS_DATA_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 750 $DFS_DATA_DIR;

[root@hadoopmaster1 hadoop]# ls -ld $DFS_DATA_DIR

drwxr-x---. 2 hdfs hadoop 4096 Sep  3 23:40 /grid/hadoop/hdfs/dn

[root@hadoopmaster1 hadoop]# mkdir -p $YARN_LOCAL_DIR;

[root@hadoopmaster1 hadoop]# chown -R $YARN_USER:$HADOOP_GROUP $YARN_LOCAL_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 755 $YARN_LOCAL_DIR;

[root@hadoopmaster1 hadoop]# mkdir -p $YARN_LOCAL_LOG_DIR;

[root@hadoopmaster1 hadoop]# chown -R $YARN_USER:$HADOOP_GROUP $YARN_LOCAL_LOG_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 755 $YARN_LOCAL_LOG_DIR;

[root@hadoopmaster1 hadoop]# mkdir -p $HDFS_LOG_DIR;

[root@hadoopmaster1 hadoop]# chown -R $HDFS_USER:$HADOOP_GROUP $HDFS_LOG_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 755 $HDFS_LOG_DIR;

[root@hadoopmaster1 hadoop]# mkdir -p $YARN_LOG_DIR;

[root@hadoopmaster1 hadoop]# chown -R $YARN_USER:$HADOOP_GROUP $YARN_LOG_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 755 $YARN_LOG_DIR;

[root@hadoopmaster1 hadoop]# mkdir -p $HDFS_PID_DIR;

[root@hadoopmaster1 hadoop]# chown -R $HDFS_USER:$HADOOP_GROUP $HDFS_PID_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 755 $HDFS_PID_DIR

[root@hadoopmaster1 hadoop]# mkdir -p $YARN_PID_DIR;

[root@hadoopmaster1 hadoop]# chown -R $YARN_USER:$HADOOP_GROUP $YARN_PID_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 755 $YARN_PID_DIR;

[root@hadoopmaster1 hadoop]# mkdir -p $MAPRED_LOG_DIR;

[root@hadoopmaster1 hadoop]# chown -R $MAPRED_USER:$HADOOP_GROUP $MAPRED_LOG_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 755 $MAPRED_LOG_DIR;

[root@hadoopmaster1 hadoop]# mkdir -p $MAPRED_PID_DIR;

[root@hadoopmaster1 hadoop]# chown -R $MAPRED_USER:$HADOOP_GROUP $MAPRED_PID_DIR;

[root@hadoopmaster1 hadoop]# chmod -R 755 $MAPRED_PID_DIR;

mkdir -p /grid/hadoop/hdfs/journal
chown -R $HDFS_USER:$HADOOP_GROUP /grid/hadoop/hdfs/journal;
chmod -R 755 /grid/hadoop/hdfs/journal

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.5/bk_installing_manually_book/content/rpm_chap3.html

cd /home/hadoop/Downloads/hdp_manual_install_rpm_helper_files-2.1.5.695/configuration_files/core_hadoop
vi core-site.xml

    <name>fs.defaultFS</name>
    <value>hdfs://hadoopmaster1:8020</value>

vi hdfs-site.xml

    <name>dfs.datanode.data.dir</name>

    <value>/grid/hadoop/hdfs/dn</value>

    <name>dfs.namenode.name.dir</name>

    <value>/grid/hadoop/hdfs/nn</value>

    <name>dfs.namenode.checkpoint.dir</name>

    <value>/grid/hadoop/hdfs/snn</value>

    <name>dfs.namenode.http-address</name>

    <value>hadoopmaster1:50070</value>

    <name>dfs.namenode.https-address</name>

     <value>hadoopmaster1:50470</value>

    <name>dfs.namenode.secondary.http-address</name>

    <value>hadoopmaster1:50090</value>

    <name>dfs.journalnode.edits.dir</name>

    <value>/grid/hadoop/hdfs/journal</value>

vi yarn-site.xml

    <name>yarn.resourcemanager.resource-tracker.address</name>

    <value>hadoopmaster1:8025</value>

    <name>yarn.resourcemanager.scheduler.address</name>

    <value>hadoopmaster1:8030</value>

    <name>yarn.resourcemanager.address</name>

    <value>hadoopmaster1:8050</value>

    <name>yarn.resourcemanager.admin.address</name>

    <value>hadoopmaster1:8141</value>

    <name>yarn.nodemanager.local-dirs</name>

    <value>/grid/hadoop/yarn/local</value>

    <name>yarn.nodemanager.log-dirs</name>

    <value>/grid/hadoop/yarn/logs</value>

    <name>yarn.log.server.url</name>

    <value>hadoopmaster1:19888/jobhistory/logs</value>

    <name>yarn.resourcemanager.webapp.address</name>

    <value>hadoopmaster1:8088</value>

vi mapred-site.xml

    <name>mapreduce.jobhistory.address</name>

      <value>hadoopmaster1:10020</value>

    <name>mapreduce.jobhistory.webapp.address</name>

      <value>hadoopmaster1:19888</value>

cd /etc/hadoop

mv conf conf.old

mkdir conf

cd /home/hadoop/Downloads/hdp_manual_install_rpm_helper_files-2.1.5.695/configuration_files/core_hadoop

cp * /etc/hadoop/conf

[root@hadoopmaster1 hadoop]# chown -R $HDFS_USER:$HADOOP_GROUP /etc/hadoop/conf

[root@hadoopmaster1 hadoop]# chmod -R 755 /etc/hadoop/conf

cd ~hadoop

cp .bash_profile ~hdfs

cp directories_profile ~hdfs

cp usersAndGroups_profile ~hdfs

cd ~hdfs

chown hdfs:hdfs .bash_profile

chown hdfs:hdfs directories_profile

chown hdfs:hdfs usersAndGroups_profile

su $HDFS_USER

. .bash_profile

$ echo $JAVA_HOME

/usr/java/default

$ which java

/usr/java/default/bin/java

$ /usr/lib/hadoop/bin/hadoop namenode -format

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

14/09/04 22:53:04 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = hadoopmaster1/192.168.1.187

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 2.4.0.2.1.5.0-695

STARTUP_MSG:   classpath = /etc/hadoop/conf:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/xz-1.0.jar:

STARTUP_MSG:   build = git@github.com:hortonworks/hadoop.git -r c11220208321e1835912fde828f1038eedb1afae; compiled by 'jenkins' on 2014-08-28T03:10Z

STARTUP_MSG:   java = 1.7.0_65

************************************************************/

14/09/04 22:53:04 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]

14/09/04 22:53:04 INFO namenode.NameNode: createNameNode [-format]

14/09/04 22:53:06 WARN common.Util: Path /grid/hadoop/hdfs/nn should be specified as a URI in configuration files. Please update hdfs configuration.

14/09/04 22:53:06 WARN common.Util: Path /grid/hadoop/hdfs/nn should be specified as a URI in configuration files. Please update hdfs configuration.

Formatting using clusterid: CID-a61ef8e5-73d6-41cd-a3d3-dde11a42344f

14/09/04 22:53:07 INFO namenode.FSNamesystem: fsLock is fair:true

14/09/04 22:53:07 INFO blockmanagement.HeartbeatManager: Setting heartbeat recheck interval to 30000 since dfs.namenode.stale.datanode.interval is less than dfs.namenode.heartbeat.recheck-interval

14/09/04 22:53:07 INFO namenode.HostFileManager: read includes:

HostSet(

)

14/09/04 22:53:07 INFO namenode.HostFileManager: read excludes:

HostSet(

)

14/09/04 22:53:07 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000

14/09/04 22:53:07 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true

14/09/04 22:53:07 INFO util.GSet: Computing capacity for map BlocksMap

14/09/04 22:53:07 INFO util.GSet: VM type       = 64-bit

14/09/04 22:53:07 INFO util.GSet: 2.0% max memory 960 MB = 19.2 MB

14/09/04 22:53:07 INFO util.GSet: capacity      = 2^21 = 2097152 entries

14/09/04 22:53:07 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=true

14/09/04 22:53:07 INFO blockmanagement.BlockManager: dfs.block.access.key.update.interval=600 min(s), dfs.block.access.token.lifetime=600 min(s), dfs.encrypt.data.transfer.algorithm=null

14/09/04 22:53:07 INFO blockmanagement.BlockManager: defaultReplication         = 3

14/09/04 22:53:07 INFO blockmanagement.BlockManager: maxReplication             = 50

14/09/04 22:53:07 INFO blockmanagement.BlockManager: minReplication             = 1

14/09/04 22:53:07 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2

14/09/04 22:53:07 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false

14/09/04 22:53:07 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000

14/09/04 22:53:07 INFO blockmanagement.BlockManager: encryptDataTransfer        = false

14/09/04 22:53:07 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000

14/09/04 22:53:07 INFO namenode.FSNamesystem: fsOwner             = hdfs (auth:SIMPLE)

14/09/04 22:53:07 INFO namenode.FSNamesystem: supergroup          = hdfs

14/09/04 22:53:07 INFO namenode.FSNamesystem: isPermissionEnabled = true

14/09/04 22:53:07 INFO namenode.FSNamesystem: HA Enabled: false

14/09/04 22:53:07 INFO namenode.FSNamesystem: Append Enabled: true

14/09/04 22:53:07 INFO util.GSet: Computing capacity for map INodeMap

14/09/04 22:53:07 INFO util.GSet: VM type       = 64-bit

14/09/04 22:53:07 INFO util.GSet: 1.0% max memory 960 MB = 9.6 MB

14/09/04 22:53:07 INFO util.GSet: capacity      = 2^20 = 1048576 entries

14/09/04 22:53:07 INFO namenode.NameNode: Caching file names occuring more than 10 times

14/09/04 22:53:07 INFO util.GSet: Computing capacity for map cachedBlocks

14/09/04 22:53:07 INFO util.GSet: VM type       = 64-bit

14/09/04 22:53:07 INFO util.GSet: 0.25% max memory 960 MB = 2.4 MB

14/09/04 22:53:07 INFO util.GSet: capacity      = 2^18 = 262144 entries

14/09/04 22:53:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 1.0

14/09/04 22:53:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0

14/09/04 22:53:07 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000

14/09/04 22:53:07 INFO namenode.FSNamesystem: Retry cache on namenode is enabled

14/09/04 22:53:07 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis

14/09/04 22:53:07 INFO util.GSet: Computing capacity for map NameNodeRetryCache

14/09/04 22:53:07 INFO util.GSet: VM type       = 64-bit

14/09/04 22:53:07 INFO util.GSet: 0.029999999329447746% max memory 960 MB = 294.9 KB

14/09/04 22:53:07 INFO util.GSet: capacity      = 2^15 = 32768 entries

14/09/04 22:53:07 INFO namenode.AclConfigFlag: ACLs enabled? false

14/09/04 22:53:08 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1235674618-192.168.1.187-1409889187946

14/09/04 22:53:08 INFO common.Storage: Storage directory /grid/hadoop/hdfs/nn has been successfully formatted.

14/09/04 22:53:08 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

14/09/04 22:53:08 INFO util.ExitUtil: Exiting with status 0

14/09/04 22:53:08 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at hadoopmaster1/192.168.1.187

************************************************************/

$ /usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start namenode

starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-hadoopmaster1.out

$ /usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start secondarynamenode

starting secondarynamenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-secondarynamenode-hadoopmaster1.out

$ /usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start datanode

starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-hadoopmaster1.out

http://192.168.1.187:50070

Create the hdfs user directory in HDFS
$ hadoop fs -mkdir -p /user/hdfs

Copy a file
$ hadoop fs -copyFromLocal /etc/passwd passwd
$ hadoop fs -ls
Found 1 items
-rw-r--r--   3 hdfs hdfs       1916 2014-09-04 23:10 passwd

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/5.0/CDH5-Security-Guide/cdh5sg_other_hadoop_security.html?scroll=topic_18_unique_2

The container-executor program must have a very specific set of permissions and ownership in order to function correctly. In particular, it must:

1. Be owned by root
2. Be owned by a group that contains only the user running the YARN daemons
3. Be setuid
4. Be group readable and executable

This corresponds to the ownership root:hadoop and the permissions 6050.

chown root:hadoop /usr/lib/hadoop-yarn/bin/container-executor

chmod 6050 /usr/lib/hadoop-yarn/bin/container-executor

[root@hadoopmaster1 yarn]# ls -al /usr/lib/hadoop-yarn/bin/container-executor

---Sr-s---. 1 root hadoop 40096 Aug 27 23:15 /usr/lib/hadoop-yarn/bin/container-executor

http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html#SingleCluster
http://hortonworks.com/community/forums/topic/permission-on-usrlibhadoop-yarnbincontainer-executor-incorrect-after-upgrad/

su yarn

$ /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager

starting resourcemanager, logging to /var/log/hadoop-yarn/yarn/yarn-yarn-resourcemanager-hadoopmaster1.out

$ /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager

starting nodemanager, logging to /var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-hadoopmaster1.out

su $HDFS_USER

hadoop fs -mkdir -p /mr-history/tmp

hadoop fs -chmod -R 1777 /mr-history/tmp

hadoop fs -mkdir -p /mr-history/done

hadoop fs -chmod -R 1777 /mr-history/done

hadoop fs -chown -R $MAPRED_USER:$HDFS_USER /mr-history

hadoop fs -mkdir -p /app-logs

hadoop fs -chmod -R 1777 /app-logs

hadoop fs -chown yarn /app-logs

su mapred

/usr/lib/hadoop-mapreduce/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver

http://192.168.1.187:8088

Browse resource manager :

su hdfs
hdfs dfs -mkdir /user/hadoop
hdfs dfs -chown hadoop:hadoop /user/hadoop
hdfs dfs -chmod -R 755 /user/hadoop

su hadoop
/usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar teragen 100 tmp/teragenout

14/09/09 19:27:00 INFO mapreduce.Job: Task Id : attempt_1410306715808_0001_m_000001_2, Status : FAILED

Error: java.net.ConnectException: Call From hadoopmaster1/192.168.1.187 to hadoopmaster1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

Need to restart HDFS

[hadoop@hadoopmaster1 ~]$ /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar teragen 10 tmp/teragenout2
14/09/09 21:25:26 INFO client.RMProxy: Connecting to ResourceManager at hadoopmaster1/192.168.1.187:8050
14/09/09 21:25:29 INFO terasort.TeraSort: Generating 10 using 2
14/09/09 21:25:29 INFO mapreduce.JobSubmitter: number of splits:2
14/09/09 21:25:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1410306715808_0002
14/09/09 21:25:32 INFO impl.YarnClientImpl: Submitted application application_1410306715808_0002
14/09/09 21:25:33 INFO mapreduce.Job: The url to track the job: http://hadoopmaster1:8088/proxy/application_1410306715808_0002/
14/09/09 21:25:33 INFO mapreduce.Job: Running job: job_1410306715808_0002
14/09/09 21:26:04 INFO mapreduce.Job: Job job_1410306715808_0002 running in uber mode : false
14/09/09 21:26:05 INFO mapreduce.Job:  map 0% reduce 0%
14/09/09 21:26:49 INFO mapreduce.Job:  map 100% reduce 0%
14/09/09 21:26:56 INFO mapreduce.Job: Job job_1410306715808_0002 completed successfully
14/09/09 21:26:57 INFO mapreduce.Job: Counters: 31
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=192632
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=158
                HDFS: Number of bytes written=1000
                HDFS: Number of read operations=8
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=4
        Job Counters
                Launched map tasks=2
                Other local map tasks=2
                Total time spent by all maps in occupied slots (ms)=264387
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=88129
                Total vcore-seconds taken by all map tasks=88129
                Total megabyte-seconds taken by all map tasks=135366144
        Map-Reduce Framework
                Map input records=10
                Map output records=10
                Input split bytes=158
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=399
                CPU time spent (ms)=3170
                Physical memory (bytes) snapshot=188051456
                Virtual memory (bytes) snapshot=2521628672
                Total committed heap usage (bytes)=31719424
        org.apache.hadoop.examples.terasort.TeraGen$Counters
                CHECKSUM=24474895181
        File Input Format Counters
                Bytes Read=0
        File Output Format Counters
                Bytes Written=1000
[hadoop@hadoopmaster1 ~]$