在使用ruby作为Map器和reduce函数运行hadoop流时,出现以下错误。
packageJobJar: [summarymapper.rb, wcreducer.rb, /var/lib/hadoop/hadoop-unjar6514686449101598265/] [] /var/folders/md/0ww65qrx1_n1nlhrr7hrs8d00000gn/T/streamjob9165241112855689376.jar tmpDir=null
14/06/25 19:54:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/06/25 19:54:35 WARN snappy.LoadSnappy: Snappy native library not loaded
14/06/25 19:54:35 INFO mapred.FileInputFormat: Total input paths to process : 1
14/06/25 19:54:35 INFO streaming.StreamJob: getLocalDirs(): [/var/lib/hadoop/mapred/local]
14/06/25 19:54:35 INFO streaming.StreamJob: Running job: job_201406251944_0005
14/06/25 19:54:35 INFO streaming.StreamJob: To kill this job, run:
14/06/25 19:54:35 INFO streaming.StreamJob: /Users/oladotunopasina/hadoop-1.2.1/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201406251944_0005
14/06/25 19:54:35 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201406251944_0005
14/06/25 19:54:36 INFO streaming.StreamJob: map 0% reduce 0%
14/06/25 19:55:18 INFO streaming.StreamJob: map 100% reduce 100%
14/06/25 19:55:18 INFO streaming.StreamJob: To kill this job, run:
14/06/25 19:55:18 INFO streaming.StreamJob: /Users/oladotunopasina/hadoop-1.2.1/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201406251944_0005
14/06/25 19:55:18 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201406251944_0005
14/06/25 19:55:18 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201406251944_0005_m_000001
14/06/25 19:55:18 INFO streaming.StreamJob: killJob...
Streaming Command Failed!
在检查生成的日志文件时,我看到
stderr logs
2014-06-25 19:54:38.332 java[8468:1003] Unable to load realm info from SCDynamicStore
env: ruby\r: No such file or directory
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:394)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
我试过使用hadoop环境变量这个线程
但仍然没有成功。请帮忙。
1条答案
按热度按时间mctunoxg1#
我通过在mac上重新保存.rb文件解决了这个问题。我下载的版本似乎保存为pc文件。”\“r”是Map器和还原器类中存在的隐藏字符。