我正在尝试使用最简单的hadoop“hello world”设置,但是当我运行以下命令时:
hadoop jar /usr/share/hadoop/hadoop-examples-1.0.4.jar grep input output 'dfs[a-z.]+'
我得到以下警告:
12/11/30 16:36:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
完整的错误跟踪如下所示:
12/11/30 16:57:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/11/30 16:57:18 WARN snappy.LoadSnappy: Snappy native library not loaded
12/11/30 16:57:18 INFO mapred.FileInputFormat: Total input paths to process : 6
12/11/30 16:57:18 INFO mapred.JobClient: Running job: job_local_0001
12/11/30 16:57:18 INFO util.ProcessTree: setsid exited with exit code 0
12/11/30 16:57:18 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7d4ecfa4
12/11/30 16:57:18 INFO mapred.MapTask: numReduceTasks: 1
12/11/30 16:57:18 INFO mapred.MapTask: io.sort.mb = 100
12/11/30 16:57:18 WARN mapred.LocalJobRunner: job_local_0001
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:949)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
12/11/30 16:57:19 INFO mapred.JobClient: map 0% reduce 0%
12/11/30 16:57:19 INFO mapred.JobClient: Job complete: job_local_0001
12/11/30 16:57:19 INFO mapred.JobClient: Counters: 0
12/11/30 16:57:19 INFO mapred.JobClient: Job Failed: NA
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
at org.apache.hadoop.examples.Grep.run(Grep.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.examples.Grep.main(Grep.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
我正在运行ubuntu 12.04和java版本:
java version "1.7.0"
Java(TM) SE Runtime Environment (build 1.7.0-b147)
Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)
有什么想法吗?
1条答案
按热度按时间9nvpjoqh1#
警告告诉您压缩编解码器没有(正确地)为hadoop安装。安装
Snappy
压缩,看看:http://code.google.com/p/hadoop-snappy/然而,一个更严重的问题是
OutOfMemoryError
你得到了。检查输入,必要时增加堆大小。你也可以看看这个相关的问题:hadoop内存不足错误