我尝试启动mapreduce作业,但在shell或配置单元中执行作业时出错:
配置单元>从员工中选择计数(*);query id=mapr\u 20171107135114\u a574713d-7d69-45e1-aa73-d4de07a3059b total jobs=1启动作业1在编译时确定的reduce任务数量:1为了更改reducer的平均负载(以字节为单位):set hive.exec.reducers.bytes.per.reducer=为了限制reducer的最大数量:set hive.exec.reducers.max=in设置常量还原数的顺序:set mapreduce.job.reduces=starting job=job\u 1510052734193\u 0005,跟踪url=http://hdpsrvpre2.intranet.darty.fr:8088/proxy/application\u 1510052734193\u 0005/kill command=/opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop job-kill job\u 1510052734193\u 0005 stage-1的hadoop作业信息:Map程序数:0;减速机数量:0 2017-11-07 13:51:25951 stage-1 map=0%,reduce=0%ended job=job\u 1510052734193\u 0005作业期间出错,获取调试信息**失败:执行错误,从org.apache.hadoop.hive.ql.exec.mr.mapredtask返回代码2 mapreduce启动的作业:stage-stage-1:maprefs读取:0 maprefs写入:0失败花费的mapreduce cpu总时间:0毫秒
在ResourceManager日志中,我发现:
> 2017-11-07 13:51:25,269 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_1510052734193_0005_000002 State change from LAUNCHED to
> FINAL_SAVING 2017-11-07 13:51:25,269 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
> Updating info for attempt: appattempt_1510052734193_0005_000002 at:
> /var/mapr/cluster/yarn/rm/system/FSRMStateRoot/RMAppRoot/application_1510052734193_0005/appattempt_1510052734193_0005_000002
> 2017-11-07 13:51:25,283 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Unregistering app attempt : appattempt_1510052734193_0005_000002
> 2017-11-07 13:51:25,283 INFO
> org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
> Application finished, removing password for
> appattempt_1510052734193_0005_000002 2017-11-07 13:51:25,283**INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_1510052734193_0005_000002 State change from FINAL_SAVING to
> FAILED**2017-11-07 13:51:25,284 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The
> number of failed attempts is 2. The max attempts is 2 2017-11-07
> 13:51:25,284 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
> Updating application application_1510052734193_0005 with final state:
> FAILED 2017-11-07 13:51:25,284 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
> application_1510052734193_0005 State change from ACCEPTED to
> FINAL_SAVING 2017-11-07 13:51:25,284 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
> Updating info for app: application_1510052734193_0005 2017-11-07
> 13:51:25,284 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> Application appattempt_1510052734193_0005_000002 is done.
> finalState=FAILED 2017-11-07 13:51:25,284 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
> Updating info for app: application_1510052734193_0005 at:
> /var/mapr/cluster/yarn/rm/system/FSRMStateRoot/RMAppRoot/application_1510052734193_0005/application_1510052734193_0005
> 2017-11-07 13:51:25,284 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo:
> Application application_1510052734193_0005 requests cleared 2017-11-07
> 13:51:25,296 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
> Application application_1510052734193_0005 failed 2 times due to AM
> Container for appattempt_1510052734193_0005_000002 exited with
> exitCode: 1 For more detailed output, check application tracking
> page:http://hdpsrvpre2.intranet.darty.fr:8088/cluster/app/application_1510052734193_0005Then,
> click on links to logs of each attempt. Diagnostics: Exception from
> container-launch. Container id:
> container_e10_1510052734193_0005_02_000001 Exit code: 1 Stack trace:
> ExitCodeException exitCode=1: at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at
> org.apache.hadoop.util.Shell.run(Shell.java:456) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:304)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:354)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:87)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
> at java.lang.Thread.run(Thread.java:748) Shell output: main : command
> provided 1 main : user is mapr main : requested yarn user is mapr
>
> Container exited with a non-zero exit code 1 Failing this attempt. Failing the application.
另外,在我找到的工作日志中:
2017-11-07 12:09:46,419 fatal[main]app.dagappmaster:启动dagappmaster java.lang.illegalargumentexception时出错:containerid无效:container\u e10\u 1510052734193\u 0001\u 01\u000001 at org.apache.hadoop.yarn.util.converterutils.tocontainerid(converterutils)。java:182)在org.apache.tez.dag.app.dagappmaster.main(dagappmaster。java:1794)原因:java.lang.numberformatexception:对于输入字符串:“e10”
在java.lang.numberformatexception.forinputstring(numberformatexception。java:65)在java.lang.long.parselong(long。java:441)在java.lang.long.parselong(long。java:483)位于org.apache.hadoop.yarn.util.converterutils.toapplicationattentid(converterutils)。java:137)在org.apache.hadoop.yarn.util.converterutils.tocontainerid(converterutils。java:177) ... 还有1个
似乎是那个问题是由tez引起的,有什么解决办法吗?谢谢您!
1条答案
按热度按时间9jyewag01#
我认为执行环境有不同版本的hadoop及其各自的jar文件。
请验证环境,确保仅使用所需版本,并从任何环境变量中删除其他版本的引用。