mapreduce作业状态一直处于运行状态

yftpprvb  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(268)

我正在尝试从oozie（4.1.0）运行mapreduce程序。
但它的状态是在运行 state 而且还停留在同样的状态。
workflow.xml文件

  1. <workflow-app xmlns="uri:oozie:workflow:0.4" name="simple-Workflow">
  2.    <start to="RunMapreduceJob" />
  3.    <action name="RunMapreduceJob">
  4.       <map-reduce>
  5.          <job-tracker>localhost:8088</job-tracker>
  6.          <name-node>hdfs://localhost:9000</name-node>
  7.          <prepare>
  8.             <delete path="hdfs://localhost:9000/dataoutput"/>
  9.          </prepare>
  10.          <configuration>
  11.             <property>
  12.                <name>mapred.job.queue.name</name>
  13.                <value>default</value>
  14.             </property>
  15.             <property>
  16.                <name>mapred.mapper.class</name>
  17.                <value>DataDividerByUser.DataDividerMapper</value>
  18.             </property>
  19.             <property>
  20.                <name>mapred.reducer.class</name>
  21.                <value>DataDividerByUser.DataDividerReducer</value>
  22.             </property>
  23.             <property>
  24.                <name>mapred.output.key.class</name>
  25.                <value>org.apache.hadoop.io.IntWritable</value>
  26.             </property>
  27.             <property>
  28.                <name>mapred.output.value.class</name>
  29.                <value>org.apache.hadoop.io.Text</value>
  30.             </property>
  31.             <property>
  32.                <name>mapred.input.dir</name>
  33.                <value>/data</value>
  34.             </property>
  35.             <property>
  36.                <name>mapred.output.dir</name>
  37.                <value>/dataoutput</value>
  38.             </property>
  39.          </configuration>
  40.       </map-reduce>
  41.       <ok to="end" />
  42.       <error to="fail" />
  43.    </action>
  44.    <kill name="fail">
  45.       <message>Mapreduce program Failed</message>
  46.    </kill>
  47.    <end name="end" />
  48. </workflow-app>

作业属性

  1. nameNode=hdfs://localhost:9000
  2. jobTracker=localhost:8088
  3. queueName=default
  4. oozie.use.system.libpath=true
  5. oozie.wf.application.path=${nameNode}/Config

job tracker也在运行，下面是一个屏幕截图 https://prnt.sc/pbvb5i 在oozie url中获取作业信息时出错

  1. JA009: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "ec2-18-222-170-204.us-east-2.compute.amazonaws.com/18.222.170.204"; destination host is: "localhost":8088;

请问发生了什么事。。？
更新：
现在所有节点都正常工作 https://prnt.sc/pc4a7n oozie原木

  1. hdfs://localhost:9000/user/hduser/share/lib/lib_20190928171545/sqoop/oozie-sharelib-sqoop-4.1.0.jar, hdfs://localhost:9000/user/hduser/share/l
  2. ib/lib_20190928171545/sqoop/sqoop-1.4.3-hadoop100.jar]
  3. 2019-09-28 17:34:29,232  INFO Services:541 - SERVER[localhost] Initialized
  4. 2019-09-28 17:34:29,234  INFO Services:541 - SERVER[localhost] Running with JARs for Hadoop version [2.3.0]
  5. 2019-09-28 17:34:29,234  INFO Services:541 - SERVER[localhost] Oozie System ID [oozie-hdus] started!
  6. 2019-09-28 17:34:29,526  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  7. 2019-09-28 17:34:29,526  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  8. 2019-09-28 17:34:29,536  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  9. 2019-09-28 17:34:29,536  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  10. 2019-09-28 17:34:29,560  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  11. 2019-09-28 17:34:29,560  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  12. 2019-09-28 17:34:29,562  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  13. 2019-09-28 17:34:29,562  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  14. 2019-09-28 17:34:39,222  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache
  15. .oozie.service.StatusTransitService]
  16. 2019-09-28 17:34:39,224  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running coordinator status se
  17. rvice first instance
  18. 2019-09-28 17:34:39,222  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTra
  19. nsitService]
  20. 2019-09-28 17:34:39,519  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running bundle status service
  21.  first instance
  22. 2019-09-28 17:34:39,521  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:541 - SERVER[localhost] USER[-] GROUP[-] TOKEN[-] 
  23. APP[-] JOB[-] ACTION[-] CoordMaterializeTriggerService - Curr Date= 2019-09-28T12:09Z, Num jobs to materialize = 0
  24. 2019-09-28 17:34:39,521  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:541 - SERVER[localhost] USER[-] GROUP[-] TOKEN[-] 
  25. APP[-] JOB[-] ACTION[-] Released lock for [org.apache.oozie.service.CoordMaterializeTriggerService]
  26. 2019-09-28 17:34:39,570  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache
  27. .oozie.service.StatusTransitService]
  28. 2019-09-28 17:34:39,571  INFO PurgeXCommand:541 - SERVER[localhost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] STARTED Purge to purge Wo
  29. rkflow Jobs older than [30] days, Coordinator Jobs older than [7] days, and Bundlejobs older than [7] days.
  30. 2019-09-28 17:34:39,571  INFO PurgeXCommand:541 - SERVER[localhost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] ENDED Purge deleted [0] w
  31. orkflows, [0] coordinatorActions, [0] coordinators, [0] bundles
  32. 2019-09-28 17:34:39,639  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTra
  33. nsitService]
  34. 2019-09-28 17:35:39,571  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache
  35. .oozie.service.StatusTransitService]
  36. 2019-09-28 17:35:39,572  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running coordinator status se
  37. rvice from last instance time =  2019-09-28T12:04Z
  38. 2019-09-28 17:35:39,616  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running bundle status service
  39.  from last instance time =  2019-09-28T12:04Z
  40. 2019-09-28 17:35:39,630  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache
  41. .oozie.service.StatusTransitService]
  42. 2019-09-28 17:35:39,647  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTra
  43. nsitService]
  44. 2019-09-28 17:35:39,662  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTra
  45. nsitService]
  46. 2019-09-28 17:35:39,814  INFO ActionStartXCommand:541 - SERVER[localhost] USER[hduser] GROUP[-] TOKEN[] APP[simple-Workflow] JOB[0000000-190928
  47. 171702728-oozie-hdus-W] ACTION[0000000-190928171702728-oozie-hdus-W@RunMapreduceJob] Start action [0000000-190928171702728-oozie-hdus-W@RunMapr
  48. educeJob] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
  49. 2019-09-28 17:36:39,631  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache
  50. .oozie.service.StatusTransitService]
  51. 2019-09-28 17:36:39,632  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running coordinator status se
  52. rvice from last instance time =  2019-09-28T12:05Z
  53. 2019-09-28 17:36:39,639  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running bundle status service
  54.  from last instance time =  2019-09-28T12:05Z
  55. 2019-09-28 17:36:39,643  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache
  56. .oozie.service.StatusTransitService]
  57. 2019-09-28 17:36:39,663  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTra
  58. nsitService]
  59. 2019-09-28 17:36:39,685  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTra
  60. nsitService]
  61. 2019-09-28 17:37:39,644  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache
  62. .oozie.service.StatusTransitService]
  63. 2019-09-28 17:37:39,645  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running coordinator status se
  64. rvice from last instance time =  2019-09-28T12:06Z
  65. 2019-09-28 17:37:39,656  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running bundle status service
  66.  from last instance time =  2019-09-28T12:06Z
  67. 2019-09-28 17:37:39,661  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache
  68. .oozie.service.StatusTransitService]
  69. 2019-09-28 17:37:39,686  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTra
  70. nsitService]
  71. 2019-09-28 17:37:39,705  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTra
  72. nsitService]
  73. 2019-09-28 17:37:53,297  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  74. 2019-09-28 17:37:53,297  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  75. 2019-09-28 17:37:53,299  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  76. 2019-09-28 17:37:53,299  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  77. 2019-09-28 17:37:53,312  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  78. 2019-09-28 17:37:53,312  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  79. 2019-09-28 17:37:53,478  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  80. 2019-09-28 17:37:53,478  WARN AuthenticationFilter:341 - SERVER[localhost] AuthenticationToken ignored: AuthenticationToken expired
  81. 2019-09-28 17:37:53,631  WARN ParameterVerifier:544 - SERVER[localhost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does 
  82. not define formal parameters in its XML definition
  83. 2019-09-28 17:37:53,893  INFO ActionStartXCommand:541 - SERVER[localhost] USER[hduser] GROUP[-] TOKEN[] APP[simple-Workflow] JOB[0000000-190928
  84. 173423962-oozie-hdus-W] ACTION[0000000-190928173423962-oozie-hdus-W@:start:] Start action [0000000-190928173423962-oozie-hdus-W@:start:] with u
  85. ser-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
  86. 2019-09-28 17:37:53,895  INFO ActionStartXCommand:541 - SERVER[localhost] USER[hduser] GROUP[-] TOKEN[] APP[simple-Workflow] JOB[0000000-190928
  87. 173423962-oozie-hdus-W] ACTION[0000000-190928173423962-oozie-hdus-W@:start:] [***0000000-190928173423962-oozie-hdus-W@:start:***]Action status=
  88. DONE
  89. 2019-09-28 17:37:53,895  INFO ActionStartXCommand:541 - SERVER[localhost] USER[hduser] GROUP[-] TOKEN[] APP[simple-Workflow] JOB[0000000-190928
  90. 173423962-oozie-hdus-W] ACTION[0000000-190928173423962-oozie-hdus-W@:start:] [***0000000-190928173423962-oozie-hdus-W@:start:***]Action updated
  91.  in DB!
  92. 2019-09-28 17:37:54,128  INFO ActionStartXCommand:541 - SERVER[localhost] USER[hduser] GROUP[-] TOKEN[] APP[simple-Workflow] JOB[0000000-190928
  93. 173423962-oozie-hdus-W] ACTION[0000000-190928173423962-oozie-hdus-W@RunMapreduceJob] Start action [0000000-190928173423962-oozie-hdus-W@RunMapr
  94. educeJob] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
  95. 2019-09-28 17:38:39,662  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache
  96. .oozie.service.StatusTransitService]
  97. 2019-09-28 17:38:39,663  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running coordinator status se
  98. rvice from last instance time =  2019-09-28T12:07Z
  99. 2019-09-28 17:38:39,671  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running bundle status service
  100.  from last instance time =  2019-09-28T12:07Z
  101. 2019-09-28 17:38:39,677  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache
  102. .oozie.service.StatusTransitService]
  103. 2019-09-28 17:38:39,706  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTra
  104. nsitService]
  105. 2019-09-28 17:38:39,722  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTra
  106. nsitService]
  107. 2019-09-28 17:39:39,527  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:541 - SERVER[localhost] USER[-] GROUP[-] TOKEN[-] 
  108. APP[-] JOB[-] ACTION[-] CoordMaterializeTriggerService - Curr Date= 2019-09-28T12:14Z, Num jobs to materialize = 0
  109. 2019-09-28 17:39:39,528  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:541 - SERVER[localhost] USER[-] GROUP[-] TOKEN[-] 
  110. APP[-] JOB[-] ACTION[-] Released lock for [org.apache.oozie.service.CoordMaterializeTriggerService]
  111. 2019-09-28 17:39:39,679  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache
  112. .oozie.service.StatusTransitService]
  113. 2019-09-28 17:39:39,680  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running coordinator status se
  114. rvice from last instance time =  2019-09-28T12:08Z
  115. 2019-09-28 17:39:39,687  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Running bundle status service
  116.  from last instance time =  2019-09-28T12:08Z
  117. 2019-09-28 17:39:39,691  INFO StatusTransitService$StatusTransitRunnable:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache
  118. .oozie.service.StatusTransitService]
  119. 2019-09-28 17:39:39,723  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTra
  120. nsitService]
  121. 2019-09-28 17:39:39,743  INFO PauseTransitService:541 - SERVER[localhost] USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTra
  122. nsitService]

但mapreduce仍处于运行准备状态。错误最终会像这样显现出来

  1. JA009: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "localhost/127.0.0.1"; destination host is: "localhost":8088;

更新-1 hadoop fs -ls hdfs://localhost:9000 输出

  1. drwxr-xr-x   - hduser supergroup          0 2019-09-28 19:44 hdfs://localhost:9000/user/hduser/oozie-hdus
  2. drwxr-xr-x   - hduser supergroup          0 2019-09-28 17:15 hdfs://localhost:9000/user/hduser/share
hadoopoozie

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com