java.lang.runtimeexception:org.apache.hadoop.hive.ql.metadata.hiveexception:org.apache.hadoop.ipc.remoteexception:java.io.ioexception:

r1wp621o  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(415)

在尝试运行配置单元查询以将数据插入配置单元外部表时，我遇到了一个问题。进程在reduce处失败。
诊断控制台消息如下：

  1. Task with the most failures(4): 
  2. -----
  3. Task ID:
  4.   task_201709171147_0059_r_000005
  6. URL:
  7.   http://localhost:50030/taskdetails.jsp?jobid=job_201709171147_0059&tipid=task_201709171147_0059_r_000005
  8. -----
  9. Diagnostic Messages for this Task:
  10. java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /tmp/hive-gms/hive_2017-09-17_15-05-39_454_6420114297268603231/_task_tmp.-mr-10007/_tmp.000005_3 could only be replicated to 0 nodes, instead of 1
  11.     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
  12.     at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
  13.     at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  14.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  15.     at java.lang.reflect.Method.invoke(Method.java:597)
  16.     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
  17.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
  18.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
  19.     at java.security.AccessController.doPrivileged(Native Method)
  20.     at javax.security.auth.Subject.doAs(Subject.java:396)
  21.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  22.     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
  24.     at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:270)
  25.     at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:522)
  26.     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
  27.     at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
  28.     at java.security.AccessController.doPrivileged(Native Method)
  29.     at javax.security.auth.Subject.doAs(Subject.java:396)
  30.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  31.     at org.apache.hadoop.mapred.Child.main(Child.java:249)
  32. Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /tmp/hive-gms/hive_2017-09-17_15-05-39_454_6420114297268603231/_task_tmp.-mr-10007/_tmp.000005_3 could only be replicated to 0 nodes, instead of 1
  33.     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
  34.     at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
  35.     at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  36.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  37.     at java.lang.reflect.Method.invoke(Method.java:597)
  38.     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
  39.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
  40.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
  41.     at java.security.AccessController.doPrivileged(Native Method)
  42.     at javax.security.auth.Subject.doAs(Subject.java:396)
  43.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  44.     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
  46.     at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:620)
  47.     at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
  48.     at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
  49.     at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1061)
  50.     at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1027)
  51.     at org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:866)
  52.     at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:751)
  53.     at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:819)
  54.     at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
  55.     at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
  56.     at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
  57.     at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
  58.     at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
  59.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.createForwardJoinObject(CommonJoinOperator.java:392)
  60.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:715)
  61.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:697)
  62.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:697)
  63.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:697)
  64.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:697)
  65.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:697)
  66.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:851)
  67.     at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:266)
  68.     at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:200)
  69.     ... 7 more
  70. Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /tmp/hive-gms/hive_2017-09-17_15-05-39_454_6420114297268603231/_task_tmp.-mr-10007/_tmp.000005_3 could only be replicated to 0 nodes, instead of 1
  71.     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
  72.     at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
  73.     at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  74.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  75.     at java.lang.reflect.Method.invoke(Method.java:597)
  76.     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
  77.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
  78.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
  79.     at java.security.AccessController.doPrivileged(Native Method)
  80.     at javax.security.auth.Subject.doAs(Subject.java:396)
  81.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  82.     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
  84.     at org.apache.hadoop.ipc.Client.call(Client.java:1113)
  85.     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
  86.     at $Proxy2.addBlock(Unknown Source)
  87.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  88.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  89.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  90.     at java.lang.reflect.Method.invoke(Method.java:597)
  91.     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
  92.     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
  93.     at $Proxy2.addBlock(Unknown Source)
  94.     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3720)
  95.     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3580)
  96.     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2783)
  97.     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3023)
  99. FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
  100. MapReduce Jobs Launched: 
  101. Job 0: Map: 60  Reduce: 16   Cumulative CPU: 1488.73 sec   HDFS Read: 15611147485 HDFS Write: 6697774964 SUCCESS
  102. Job 1: Map: 41  Reduce: 11   Cumulative CPU: 2231.76 sec   HDFS Read: 10853652873 HDFS Write: 8893984114 SUCCESS
  103. Job 2: Map: 31  Reduce: 10   Cumulative CPU: 2426.38 sec   HDFS Read: 9165529191 HDFS Write: 8927047843 SUCCESS
  104. Job 3: Map: 33  Reduce: 10   Cumulative CPU: 2469.67 sec   HDFS Read: 9198570523 HDFS Write: 9003617537 SUCCESS
  105. Job 4: Map: 33  Reduce: 10   Cumulative CPU: 2519.58 sec   HDFS Read: 9275146392 HDFS Write: 9032825573 SUCCESS
  106. Job 5: Map: 33  Reduce: 10   Cumulative CPU: 2607.76 sec   HDFS Read: 9304360085 HDFS Write: 9107047528 SUCCESS
  107. Job 6: Map: 34  Reduce: 10   Cumulative CPU: 2624.55 sec   HDFS Read: 9378577538 HDFS Write: 15269385865 FAIL
  108. Total MapReduce CPU Time Spent: 0 days 4 hours 32 minutes 48 seconds 430 msec

来自jobtracker web ui的日志消息如下：

  1. java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /tmp/hive-gms/hive_2017-09-17_15-05-39_454_6420114297268603231/_task_tmp.-mr-10007/_tmp.000004_0 could only be replicated to 0 nodes, instead of 1
  2.     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
  3.     at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
  4.     at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  5.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  6.     at java.lang.reflect.Method.invoke(Method.java:597)
  7.     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
  8.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
  9.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
  10.     at java.security.AccessController.doPrivileged(Native Method)
  11.     at javax.security.auth.Subject.doAs(Subject.java:396)
  12.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  13.     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
  15.     at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:270)
  16.     at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:522)
  17.     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
  18.     at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
  19.     at java.security.AccessController.doPrivileged(Native Method)
  20.     at javax.security.auth.Subject.doAs(Subject.java:396)
  21.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  22.     at org.apache.hadoop.mapred.Child.main(Child.java:249)
  23. Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /tmp/hive-gms/hive_2017-09-17_15-05-39_454_6420114297268603231/_task_tmp.-mr-10007/_tmp.000004_0 could only be replicated to 0 nodes, instead of 1
  24.     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
  25.     at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
  26.     at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  27.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  28.     at java.lang.reflect.Method.invoke(Method.java:597)
  29.     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
  30.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
  31.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
  32.     at java.security.AccessController.doPrivileged(Native Method)
  33.     at javax.security.auth.Subject.doAs(Subject.java:396)
  34.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  35.     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
  37.     at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:620)
  38.     at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
  39.     at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
  40.     at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1061)
  41.     at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:885)
  42.     at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:753)
  43.     at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:819)
  44.     at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
  45.     at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
  46.     at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
  47.     at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
  48.     at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
  49.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.createForwardJoinObject(CommonJoinOperator.java:392)
  50.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:715)
  51.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:697)
  52.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:697)
  53.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:697)
  54.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:697)
  55.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:697)
  56.     at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:851)
  57.     at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:266)
  58.     at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:200)
  59.     ... 7 more
  60. Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /tmp/hive-gms/hive_2017-09-17_15-05-39_454_6420114297268603231/_task_tmp.-mr-10007/_tmp.000004_0 could only be replicated to 0 nodes, instead of 1
  61.     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
  62.     at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
  63.     at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
  64.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  65.     at java.lang.reflect.Method.invoke(Method.java:597)
  66.     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
  67.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
  68.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
  69.     at java.security.AccessController.doPrivileged(Native Method)
  70.     at javax.security.auth.Subject.doAs(Subject.java:396)
  71.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  72.     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
  74.     at org.apache.hadoop.ipc.Client.call(Client.java:1113)
  75.     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
  76.     at $Proxy2.addBlock(Unknown Source)
  77.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  78.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  79.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  80.     at java.lang.reflect.Method.invoke(Method.java:597)
  81.     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
  82.     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
  83.     at $Proxy2.addBlock(Unknown Source)
  84.     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3720)
  85.     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3580)
  86.     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2783)
  87.     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3023)
  89.     Last 4KB
  90. Last 8KB
  91. All
  92. attempt_201709171147_0059_r_000005_0    task_201709171147_0059_r_000005 gmdatal01-p1.ilcb.tcprod.local  FAILED

我已经为配置单元设置了动态分区：

  1. set hive.exec.dynamic.partitions = true
  2. set hive.exec.dynamic.partitions.mode = nonstrict

我还格式化了namenode，并再次尝试，但错误不会消失。

hadoopHivehiveql

1条答案

按热度按时间
fhg3lkii

fhg3lkii1#

问题的根本原因似乎与空间有关。我使用了一个连接几个大表的查询，由于数据是倾斜的，hive开始创建大的临时表，因为磁盘上没有空间。我将配置文件恢复为默认值，并压缩中间结果（配置文件）以获得结果。

赞(0）分享  回复(0）举报  2021-05-29
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com