Tuesday, September 9, 2014

Hadoop : MapReduce : port 8020 failed on connection exception

Error when running MapReduce :

[hadoop@hadoopmaster1 ~]$ /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar teragen 100 tmp/teragenout
:
:
:

Caused by: java.net.ConnectException: Call From hadoopmaster1/192.168.1.187 to hadoopmaster1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused


Solution : Restart HDFS Name Node

[hdfs@hadoopmaster1 ~]$jps
32192 DataNode


su hdfs

stop
/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop namenode

/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop secondarynamenode

/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop datanode

start
/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start namenode

/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start secondarynamenode

/usr/lib/hadoop/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR start datanode


[hdfs@hadoopmaster1 ~]$jps
32192 DataNode
31851 NameNode

[root@hadoopmaster1 conf]# netstat -lent | grep :8020
tcp        0      0 192.168.1.187:8020          0.0.0.0:*                   LISTEN      493        548347

[hadoop@hadoopmaster1 ~]$ /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar teragen 10 tmp/teragenout
java.io.IOException: Output directory tmp/teragenout already exists.

[hadoop@hadoopmaster1 ~]$ /usr/lib/hadoop/bin/hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar teragen 10 tmp/teragenout2
14/09/09 21:25:26 INFO client.RMProxy: Connecting to ResourceManager at hadoopmaster1/192.168.1.187:8050
14/09/09 21:25:29 INFO terasort.TeraSort: Generating 10 using 2
14/09/09 21:25:29 INFO mapreduce.JobSubmitter: number of splits:2
14/09/09 21:25:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1410306715808_0002
14/09/09 21:25:32 INFO impl.YarnClientImpl: Submitted application application_1410306715808_0002
14/09/09 21:25:33 INFO mapreduce.Job: The url to track the job: http://hadoopmaster1:8088/proxy/application_1410306715808_0002/
14/09/09 21:25:33 INFO mapreduce.Job: Running job: job_1410306715808_0002
14/09/09 21:26:04 INFO mapreduce.Job: Job job_1410306715808_0002 running in uber mode : false
14/09/09 21:26:05 INFO mapreduce.Job:  map 0% reduce 0%
14/09/09 21:26:49 INFO mapreduce.Job:  map 100% reduce 0%
14/09/09 21:26:56 INFO mapreduce.Job: Job job_1410306715808_0002 completed successfully
14/09/09 21:26:57 INFO mapreduce.Job: Counters: 31
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=192632
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=158
                HDFS: Number of bytes written=1000
                HDFS: Number of read operations=8
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=4
        Job Counters
                Launched map tasks=2
                Other local map tasks=2
                Total time spent by all maps in occupied slots (ms)=264387
                Total time spent by all reduces in occupied slots (ms)=0
                Total time spent by all map tasks (ms)=88129
                Total vcore-seconds taken by all map tasks=88129
                Total megabyte-seconds taken by all map tasks=135366144
        Map-Reduce Framework
                Map input records=10
                Map output records=10
                Input split bytes=158
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=399
                CPU time spent (ms)=3170
                Physical memory (bytes) snapshot=188051456
                Virtual memory (bytes) snapshot=2521628672
                Total committed heap usage (bytes)=31719424
        org.apache.hadoop.examples.terasort.TeraGen$Counters
                CHECKSUM=24474895181
        File Input Format Counters
                Bytes Read=0
        File Output Format Counters
                Bytes Written=1000

17 comments:

  1. I really like the dear information you offer in your articles. I’m able to bookmark your site and show the kids check out up here generally. Im fairly positive theyre likely to be informed a great deal of new stuff here than anyone
    angularjs-Training in sholinganallur

    angularjs-Training in velachery

    angularjs Training in bangalore

    angularjs Training in bangalore

    angularjs Training in btm

    angularjs Training in electronic-city

    ReplyDelete
  2. Good Post, I am a big believer in posting comments on sites to let the blog writers know that they ve added something advantageous to the world wide web.
    online Python training
    python training in chennai

    ReplyDelete
  3. Inspiring writings and I greatly admired what you have to say , I hope you continue to provide new ideas for us all and greetings success always for you..Keep update more information.

    rpa training in chennai
    rpa training in bangalore
    rpa course in bangalore
    best rpa training in bangalore
    rpa online training

    ReplyDelete
  4. Informative post indeed, I’ve being in and out reading posts regularly and I see alot of engaging people sharing things and majority of the shared information is very valuable and so, here’s my fine read.
    click here to check results
    click here to check marks
    click here to change your password
    click here to claim your ecard
    click here to continue your profile creation

    ReplyDelete
  5. Greetings. I know this is somewhat off-topic, but I was wondering if you knew where I could get a captcha plugin for my comment form? I’m using the same blog platform like yours, and I’m having difficulty finding one?Thanks!!!

    Android Training in Chennai

    Android Online Training in Chennai

    Android Training in Bangalore

    Android Training in Hyderabad

    Android Training in Coimbatore

    Android Training

    Android Online Training

    ReplyDelete
  6. Thanks for sharing this article about seo. i read your blog every time.. oracle training in chennai

    ReplyDelete
  7. Too Good article,Thank you for sharing this awesome Blog.
    Keep Updating more....

    Big Data Hadoop Course

    ReplyDelete