hadoop示例作业在独立模式下失败:“无法加载本机hadoop库”

iugsix8n  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(380)

我正在尝试使用最简单的hadoop“hello world”设置,但是当我运行以下命令时:

hadoop jar /usr/share/hadoop/hadoop-examples-1.0.4.jar grep input output 'dfs[a-z.]+'

我得到以下警告:

12/11/30 16:36:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

完整的错误跟踪如下所示:

12/11/30 16:57:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/11/30 16:57:18 WARN snappy.LoadSnappy: Snappy native library not loaded
12/11/30 16:57:18 INFO mapred.FileInputFormat: Total input paths to process : 6
12/11/30 16:57:18 INFO mapred.JobClient: Running job: job_local_0001
12/11/30 16:57:18 INFO util.ProcessTree: setsid exited with exit code 0
12/11/30 16:57:18 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7d4ecfa4
12/11/30 16:57:18 INFO mapred.MapTask: numReduceTasks: 1
12/11/30 16:57:18 INFO mapred.MapTask: io.sort.mb = 100
12/11/30 16:57:18 WARN mapred.LocalJobRunner: job_local_0001
java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:949)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
12/11/30 16:57:19 INFO mapred.JobClient:  map 0% reduce 0%
12/11/30 16:57:19 INFO mapred.JobClient: Job complete: job_local_0001
12/11/30 16:57:19 INFO mapred.JobClient: Counters: 0
12/11/30 16:57:19 INFO mapred.JobClient: Job Failed: NA
java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
    at org.apache.hadoop.examples.Grep.run(Grep.java:69)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.examples.Grep.main(Grep.java:93)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

我正在运行ubuntu 12.04和java版本:

java version "1.7.0"
Java(TM) SE Runtime Environment (build 1.7.0-b147)
Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)

有什么想法吗?

9nvpjoqh

9nvpjoqh1#

警告告诉您压缩编解码器没有(正确地)为hadoop安装。安装 Snappy 压缩,看看:http://code.google.com/p/hadoop-snappy/
然而,一个更严重的问题是 OutOfMemoryError 你得到了。检查输入,必要时增加堆大小。你也可以看看这个相关的问题:
hadoop内存不足错误

相关问题