hadoop数据节点启动失败

f45qwnt8  于 2021-05-30  发布在  Hadoop
关注(0)|答案(4)|浏览(480)

我有一个由11个节点组成的hadoop集群。一个节点充当主节点，10个从节点运行datanode和tasktrackers。
在所有从属节点上启动任务跟踪器。datanode只启动了以下10个节点中的6个是来自 /hadoop/logs/...Datanode....log 文件。

  1. 2014-12-03 17:55:05,057 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: 
  3. /************************************************************
  5. STARTUP_MSG: Starting DataNode
  7. STARTUP_MSG:   host = trans9/192.168.0.16
  9. STARTUP_MSG:   args = []
  11. STARTUP_MSG:   version = 1.2.1
  13. STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 
  14. 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
  16. STARTUP_MSG:   java = 1.7.0_65
  18. ************************************************************/
  20. 2014-12-03 17:55:05,371 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
  22. 2014-12-03 17:55:05,384 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
  24. 2014-12-03 17:55:05,385 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
  26. 2014-12-03 17:55:05,385 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
  28. 2014-12-03 17:55:05,776 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
  30. 2014-12-03 17:55:05,789 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
  32. 2014-12-03 17:55:08,850 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
  34. 2014-12-03 17:55:08,865 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened data transfer server at 50010
  36. 2014-12-03 17:55:08,867 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
  38. 2014-12-03 17:55:08,876 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
  40. 2014-12-03 17:55:08,962 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
  42. 2014-12-03 17:55:09,055 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
  44. 2014-12-03 17:55:09,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dfs.webhdfs.enabled = false
  46. 2014-12-03 17:55:09,068 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
  48. 2014-12-03 17:55:09,068 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
  50. 2014-12-03 17:55:09,068 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
  52. 2014-12-03 17:55:09,068 INFO org.mortbay.log: jetty-6.1.26
  54. 2014-12-03 17:55:09,804 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
  56. 2014-12-03 17:55:09,812 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
  58. 2014-12-03 17:55:09,813 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source DataNode registered.
  60. 2014-12-03 17:55:09,893 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort50020 registered.
  62. 2014-12-03 17:55:09,894 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort50020 registered.
  64. 2014-12-03 17:55:09,895 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
  66. 2014-12-03 17:55:09,903 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(slave9:50010, storageID=DS-551911532-192.168.0.31-50010-1417617118848, infoPort=50075, ipcPort=50020)
  68. 2014-12-03 17:55:09,914 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Finished generating blocks being written report for 1 volumes in 0 seconds
  70. 2014-12-03 17:55:09,933 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Finished asynchronous block report scan in 5ms
  72. 2014-12-03 17:55:09,933 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.0.16:50010, storageID=DS-551911532-192.168.0.31-50010-1417617118848, infoPort=50075, ipcPort=50020)In DataNode.run, data = FSDataset{dirpath='/home/ubuntu/hadoop/hadoop-data/dfs/data/current'}
  74. 2014-12-03 17:55:09,945 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
  76. 2014-12-03 17:55:09,946 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
  78. 2014-12-03 17:55:09,946 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 50020: starting
  80. 2014-12-03 17:55:09,955 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 50020: starting
  82. 2014-12-03 17:55:09,955 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: using BLOCKREPORT_INTERVAL of 3600000msec Initial delay: 0msec
  84. 2014-12-03 17:55:09,959 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 50020: starting
  86. 2014-12-03 17:55:10,140 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is shutting down: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Data node 192.168.0.16:50010 is attempting to report storage ID DS-551911532-192.168.0.31-50010-1417617118848. Node 192.168.0.14:50010 is expected to serve this storage.
  87.     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:5049)
  88.     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport(FSNamesystem.java:3939)
  89.     at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport(NameNode.java:1095)
  90.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  91.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  92.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  93.     at java.lang.reflect.Method.invoke(Method.java:606)
  94.     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
  95.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
  96.     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
  97.     at java.security.AccessController.doPrivileged(Native Method)
  98.     at javax.security.auth.Subject.doAs(Subject.java:415)
  99.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
  100.     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)
  102.     at org.apache.hadoop.ipc.Client.call(Client.java:1113)
  103.     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
  104.     at com.sun.proxy.$Proxy3.blockReport(Unknown Source)
  105.     at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1084)
  106.     at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1588)
  107.     at java.lang.Thread.run(Thread.java:745)
  109. 2014-12-03 17:55:10,144 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:50075
  111. 2014-12-03 17:55:10,147 INFO org.apache.hadoop.ipc.Server: Stopping server on 50020
  113. 2014-12-03 17:55:10,147 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 50020: exiting
  115. 2014-12-03 17:55:10,147 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 50020: exiting
  117. 2014-12-03 17:55:10,147 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 50020: exiting
  119. 2014-12-03 17:55:10,148 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 50020
  121. 2014-12-03 17:55:10,148 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
  123. 2014-12-03 17:55:10,149 INFO org.apache.hadoop.ipc.metrics.RpcInstrumentation: shut down
  125. 2014-12-03 17:55:10,149 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.0.16:50010, storageID=DS-551911532-192.168.0.31-50010-1417617118848, infoPort=50075, ipcPort=50020):DataXceiveServer:java.nio.channels.AsynchronousCloseException
  126.     at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
  127.     at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:248)
  128.     at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100)
  129.     at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:132)
  130.     at java.lang.Thread.run(Thread.java:745)
  132. 2014-12-03 17:55:10,149 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting DataXceiveServer
  134. 2014-12-03 17:55:10,149 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 0
  136. 2014-12-03 17:55:10,150 INFO org.apache.hadoop.hdfs.server.datanode.FSDatasetAsyncDiskService: Shutting down all async disk service threads...
  138. 2014-12-03 17:55:10,151 INFO org.apache.hadoop.hdfs.server.datanode.FSDatasetAsyncDiskService: All async disk service threads have been shut down
  140. 2014-12-03 17:55:10,151 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.0.16:50010, storageID=DS-551911532-192.168.0.31-50010-1417617118848, infoPort=50075, ipcPort=50020):Finishing DataNode in: FSDataset{dirpath='/home/ubuntu/hadoop/hadoop-data/dfs/data/current'}
  142. 2014-12-03 17:55:10,152 WARN org.apache.hadoop.metrics2.util.MBeans: Hadoop:service=DataNode,name=DataNodeInfo
  143. javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=DataNodeInfo
  144.     at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
  145.     at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
  146.     at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
  147.     at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
  148.     at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:71)
  149.     at org.apache.hadoop.hdfs.server.datanode.DataNode.unRegisterMXBean(DataNode.java:586)
  150.     at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:855)
  151.     at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1601)
  152.     at java.lang.Thread.run(Thread.java:745)
  154. 2014-12-03 17:55:10,152 INFO org.apache.hadoop.ipc.Server: Stopping server on 50020
  156. 2014-12-03 17:55:10,152 INFO org.apache.hadoop.ipc.metrics.RpcInstrumentation: shut down
  158. 2014-12-03 17:55:10,153 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 0
  160. 2014-12-03 17:55:10,153 WARN org.apache.hadoop.metrics2.util.MBeans: Hadoop:service=DataNode,name=FSDatasetState-DS-551911532-192.168.0.31-50010-1417617118848
  161. javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=FSDatasetState-DS-551911532-192.168.0.31-50010-1417617118848
  162.     at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
  163.     at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
  164.     at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
  165.     at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
  166.     at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:71)
  167.     at org.apache.hadoop.hdfs.server.datanode.FSDataset.shutdown(FSDataset.java:2093)
  168.     at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:917)
  169.     at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1601)
  170.     at java.lang.Thread.run(Thread.java:745)
  172. 2014-12-03 17:55:10,159 WARN org.apache.hadoop.hdfs.server.datanode.FSDatasetAsyncDiskService: AsyncDiskService has already shut down.
  174. 2014-12-03 17:55:10,159 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
  176. 2014-12-03 17:55:10,166 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
  177. /************************************************************
  179. SHUTDOWN_MSG: Shutting down DataNode at trans9/192.168.0.16
  181. ************************************************************/
hadoopNodesstartupfailovercluster

4条答案

按热度按时间
wsxa1bj1

wsxa1bj11#

因为您的集群有几十个节点，所以使用一个节点就可以了 Namenode 以及 JobTracker . Secondary Namenode 应该在另一个节点上，因为执行定期备份需要更多内存。
说到你的问题，
可能是配置文件复制导致冲突。此处回答了相同的问题
您可以尝试通过适当的更改复制工作数据节点配置。
如果集群上没有数据，可以格式化namenode，在此之前停止所有守护进程，然后重新启动。
希望有帮助！

赞(0）分享  回复(0）举报  2021-05-30
aiqt4smr

aiqt4smr2#

这是由于不兼容的集群id问题造成的，所以请格式化datanode目录并重新开始。

赞(0）分享  回复(0）举报  2021-05-30
uubf1zoe

uubf1zoe3#

解决办法如下。这不会导致任何数据删除，因为您会将复制因子设置为大于1。解决方法是检查盒子上运行的服务。在这种情况下，您将 tasktracker 在箱子上跑。通过运行停止任务跟踪器 hadoop-daemon stop tasktracker 完成后，转到您可能在下面提到的位置 dfs.data.dir 属性并删除此处的所有文件\文件夹。完成后，运行
hadoop-daemon start datanode hadoop-daemon start tasktracker 这将启动datanode和tasktracker。如果您成功地做到了这一点，请转到namenode并运行 hadoop dfsadmin -refreshNodes

赞(0）分享  回复(0）举报  2021-05-30
ndasle7k

ndasle7k4#

哦，一个系统的名称节点和其他系统分配给数据节点是不推荐的。我推荐一个系统用于name node，另一个系统用于job tracker，另一个系统用于secondary name node。其余的8个系统应该数据节点是强烈建议。
来回答你的问题吧。格式化名称节点以解决此问题。同时重新打开终端。有的时候网络也是原因之一。

赞(0）分享  回复(0）举报  2021-05-30
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com