Big Data and Cloud

Friday, September 12, 2014

Linux : SAN : How to find out the shared SAN LUN

Problem : We have LUN presented to 2 Red Hat servers, how can we verify which device file is shared device ?

Solution : Find out the LUN with same serial number by running sq_inq

In server A :

# fdisk -l

Disk /dev/mapper/3600a098044316b37305d44353075674d: 537.0 GB, 536952700928 bytes
255 heads, 63 sectors/track, 65280 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Disk identifier: 0x00000000

# sg_inq /dev/mapper/3600a098044316b37305d44353075674d
standard INQUIRY:
PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3]
[AERC=0] [TrmTsk=0] NormACA=1 HiSUP=1 Resp_data_format=2
SCCS=0 ACC=0 TPGS=1 3PC=1 Protect=0 BQue=0
EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0
[RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1
[SPI: Clocking=0x0 QAS=0 IUS=0]
length=117 (0x75) Peripheral device type: disk
Vendor identification: NETAPP
Product identification: LUN C-Mode
Product revision level: 8200
Unit serial number: D1k70]D50ugM

In Server B :

# fdisk -l

Disk /dev/mapper/3600a098044316b37305d44353075674e: 537.0 GB, 536952700928 bytes
255 heads, 63 sectors/track, 65280 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 65536 bytes
Disk identifier: 0x00000000

# sg_inq /dev/mapper/3600a098044316b37305d44353075674e
standard INQUIRY:
PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3]
[AERC=0] [TrmTsk=0] NormACA=1 HiSUP=1 Resp_data_format=2
SCCS=0 ACC=0 TPGS=1 3PC=1 Protect=0 BQue=0
EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0
[RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1
[SPI: Clocking=0x0 QAS=0 IUS=0]
length=117 (0x75) Peripheral device type: disk
Vendor identification: NETAPP
Product identification: LUN C-Mode
Product revision level: 8200
Unit serial number: D1k70]D50ugN

Thursday, September 11, 2014

Solaris 11 : cannot ssh into the Solaris 11 server

Problem : cannot ssh into the Solaris 11 server

$ ssh id@$ip_solaris11
ssh_exchange_identification: Connection closed by remote host

Solution : vi /etc/hosts.allow, add the subnet or IP of the client and then restart inetd

vi /etc/hosts.allow and make sure the subnet or IP of the client is added to the sshd ALLOW section.

sshd:   x.x.x.x/255.255.255.0 \
        10.100.0.0/255.255.0.0 \
        x.x.x.x/255.255.255.0                       : ALLOW

Make sure sshd service is ON
# svcs -a | grep ssh
online         Sep_02   svc:/network/ssh:default

Now restart inetd
# svcadm restart inetd

Big Data Talks

http://www.kdnuggets.com/2014/09/most-viewed-big-data-talks-videolectures.html

http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation

Mining of massive datasets

http://www.mmds.org/

https://www.linkedin.com/pulse/article/20140918162814-111366377-let-s-get-nerdy-data-analytics-for-business-leaders-explained

http://simplystatistics.org/2014/05/22/10-things-statistics-taught-us-about-big-data-analysis/

http://www.datasciencecentral.com/profiles/blogs/data-science-cheat-sheet

www.coreservlets.com/hadoop-tutorial/

http://www.stratapps.net/intro-hive.php

Wednesday, September 10, 2014

Hadoop : Sqoop

http://www.slideshare.net/huguk/140410-huguksqoopunlockhadoop

https://www.youtube.com/watch?v=puQMQxkDNtw

Solaris : How To Find Fiber Host Bus Adapter (Fiber HBA) WWN

see also link

# luxadm -e port
/devices/pci@300/pci@1/pci@0/pci@4/SUNW,emlxs@0/fp@0,0:devctl      CONNECTED
/devices/pci@300/pci@1/pci@0/pci@4/SUNW,emlxs@0,1/fp@0,0:devctl    CONNECTED

# fcinfo hba-port
HBA Port WWN: 10000090fa72ac46
        Port Mode: Initiator
        Port ID: 210760
        OS Device Name: /dev/cfg/c5
        Manufacturer: Emulex
        Model: LPe12002-S
        Firmware Version: LPe12002-S 1.00a12
        FCode/BIOS Version: Boot:5.03a0 Fcode:3.01a1
        Serial Number: 4925382+1422000085
        Driver Name: emlxs
        Driver Version: 2.90.15.0 (2014.01.22.14.50)
        Type: N-port
        State: online
        Supported Speeds: 2Gb 4Gb 8Gb
        Current Speed: 8Gb
        Node WWN: 20000090fa72ac46
        NPIV Not Supported
HBA Port WWN: 10000090fa72ac47
        Port Mode: Initiator
        Port ID: 210680
        OS Device Name: /dev/cfg/c6
        Manufacturer: Emulex
        Model: LPe12002-S
        Firmware Version: LPe12002-S 1.00a12
        FCode/BIOS Version: Boot:5.03a0 Fcode:3.01a1
        Serial Number: 4925382+1422000085
        Driver Name: emlxs
        Driver Version: 2.90.15.0 (2014.01.22.14.50)
        Type: N-port
        State: online
        Supported Speeds: 2Gb 4Gb 8Gb
        Current Speed: 8Gb
        Node WWN: 20000090fa72ac47
        NPIV Not Supported

Tuesday, September 9, 2014

Hadoop : MapReduce : port 8020 failed on connection exception

Error when running MapReduce :

[hadoop@hadoopmaster1 ~]$ /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar teragen 100 tmp/teragenout

Caused by: java.net.ConnectException: Call From hadoopmaster1/192.168.1.187 to hadoopmaster1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

Solution : Restart HDFS Name Node

[hdfs@hadoopmaster1 ~]$jps
32192 DataNode

su hdfs

stop

/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop namenode

/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop secondarynamenode

/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop datanode

start

/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start namenode

/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start secondarynamenode

/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start datanode

[hdfs@hadoopmaster1 ~]$jps

32192 DataNode

31851 NameNode

[root@hadoopmaster1 conf]# netstat -lent | grep :8020

tcp 0 0 192.168.1.187:8020 0.0.0.0:* LISTEN 493 548347

[hadoop@hadoopmaster1 ~]$ /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar teragen 10 tmp/teragenout

java.io.IOException: Output directory tmp/teragenout already exists.

[hadoop@hadoopmaster1 ~]$ /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar teragen 10 tmp/teragenout2

14/09/09 21:25:26 INFO client.RMProxy: Connecting to ResourceManager at hadoopmaster1/192.168.1.187:8050

14/09/09 21:25:29 INFO terasort.TeraSort: Generating 10 using 2

14/09/09 21:25:29 INFO mapreduce.JobSubmitter: number of splits:2

14/09/09 21:25:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1410306715808_0002

14/09/09 21:25:32 INFO impl.YarnClientImpl: Submitted application application_1410306715808_0002

14/09/09 21:25:33 INFO mapreduce.Job: The url to track the job: http://hadoopmaster1:8088/proxy/application_1410306715808_0002/

14/09/09 21:25:33 INFO mapreduce.Job: Running job: job_1410306715808_0002

14/09/09 21:26:04 INFO mapreduce.Job: Job job_1410306715808_0002 running in uber mode : false

14/09/09 21:26:05 INFO mapreduce.Job: map 0% reduce 0%

14/09/09 21:26:49 INFO mapreduce.Job: map 100% reduce 0%

14/09/09 21:26:56 INFO mapreduce.Job: Job job_1410306715808_0002 completed successfully

14/09/09 21:26:57 INFO mapreduce.Job: Counters: 31

File System Counters

FILE: Number of bytes read=0

FILE: Number of bytes written=192632

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=158

HDFS: Number of bytes written=1000

HDFS: Number of read operations=8

HDFS: Number of large read operations=0

HDFS: Number of write operations=4

Job Counters

Launched map tasks=2

Other local map tasks=2

Total time spent by all maps in occupied slots (ms)=264387

Total time spent by all reduces in occupied slots (ms)=0

Total time spent by all map tasks (ms)=88129

Total vcore-seconds taken by all map tasks=88129

Total megabyte-seconds taken by all map tasks=135366144

Map-Reduce Framework

Map input records=10

Map output records=10

Input split bytes=158

Spilled Records=0

Failed Shuffles=0

Merged Map outputs=0

GC time elapsed (ms)=399

CPU time spent (ms)=3170

Physical memory (bytes) snapshot=188051456

Virtual memory (bytes) snapshot=2521628672

Total committed heap usage (bytes)=31719424

org.apache.hadoop.examples.terasort.TeraGen$Counters

CHECKSUM=24474895181

File Input Format Counters

Bytes Read=0

File Output Format Counters

Bytes Written=1000

Hadoop : Fix ERROR check containerexecutor

Where is this error coming from ?

Now checking log file :
[root@hadoopmaster1 yarn]# pwd
/var/log/hadoop-yarn/yarn
[root@hadoopmaster1 yarn]# grep ERROR yarn-yarn-resourcemanager-hadoopmaster1.log
2014-09-08 23:47:58,974 INFO rmnode.RMNodeImpl (RMNodeImpl.java:transition(621)) - Node hadoopmaster1:45454 reported UNHEALTHY with details: ERROR check containerexecutor, OK: disks ok,

In /etc/hadoop/conf/yarn-site.xml

<name>yarn.nodemanager.health-checker.script.path</name>
<value>/etc/hadoop/conf/health_check</value>

[root@hadoopmaster1 conf]# pwd
/etc/hadoop/conf
[root@hadoopmaster1 conf]# ./health_check
ERROR check containerexecutor, OK: disks ok,

in /etc/hadoop/conf/health_check

function check_containerexecutor {
perm=`stat -c %a:%U:%G /usr/lib/hadoop-yarn/bin/container-executor 2>/dev/null
`
if [ $? -eq 0 ] && [ "$perm" == "6050:root:hadoop" ] ; then
echo "containerexecutor ok"
else
echo 'check containerexecutor' ; exit 1
fi
}

Solution :
chown root:hadoop /usr/lib/hadoop-yarn/bin/container-executor
chmod 6050 /usr/lib/hadoop-yarn/bin/container-executor

[root@hadoopmaster1 yarn]# chown root:hadoop /usr/lib/hadoop-yarn/bin/container-executor
[root@hadoopmaster1 yarn]# ls -al /usr/lib/hadoop-yarn/bin/container-executor
----r-x---. 1 root hadoop 40096 Aug 27 23:15 /usr/lib/hadoop-yarn/bin/container-executor
[root@hadoopmaster1 yarn]# chmod 6050 /usr/lib/hadoop-yarn/bin/container-executor
[root@hadoopmaster1 yarn]# ls -al /usr/lib/hadoop-yarn/bin/container-executor
---Sr-s---. 1 root hadoop 40096 Aug 27 23:15 /usr/lib/hadoop-yarn/bin/container-executor

[root@hadoopmaster1 conf]# ./health_check
OK: disks ok,containerexecutor ok,

Now restart YARN

stop:
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager

/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager

start:
/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager

/usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager

Friday, September 12, 2014

Thursday, September 11, 2014

Wednesday, September 10, 2014

Tuesday, September 9, 2014

Labels