hadoop 只能写入到1个minReplication节点中的0个,有3个数据节点正在运行，此操作中排除了3个节点,正在使用Docker

lmyy7pcs  于 2022-12-11  发布在  Hadoop
关注(0)|答案(1)|浏览(406)

我制作了一个docker-compose.yml文件，其中包含Hadoop、Kafka和Flume等多个apache服务。目前，我尝试使用Kafka检索数据，并将其发送到Flume（能够转换数据（结构），并将其存储在HDFS中。我通过使用Kafka生产者生成虚拟数据，在那里我可以向Kafka代理发送消息。Flume监听某个主题，转换并定义数据的位置，并尝试将其发送到HDFS。每当flume代理注意到数据正在进入时，就会发生以下错误：

  1. 2021-11-14 20:16:13,554 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  2. 2021-11-14 20:16:17,448 WARN hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073742188_1365
  3. java.net.ConnectException: Connection refused
  4.     at java.base/sun.nio.ch.Net.pollConnect(Native Method)
  5.     at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
  6.     at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
  7.     at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
  8.     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:586)
  9.     at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
  10.     at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1757)
  11.     at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1711)
  12.     at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:707)
  13. 2021-11-14 20:16:17,451 WARN hdfs.DataStreamer: Abandoning BP-2051009381-192.168.160.8-1635954925420:blk_1073742188_1365
  14. 2021-11-14 20:16:17,462 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[192.168.192.12:50010,DS-0eb49c38-45e0-46bb-be71-23f07b5ac9dc,DISK]
  15. 2021-11-14 20:16:28,525 WARN hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073742189_1366
  16. java.net.ConnectException: Connection refused
  17.     at java.base/sun.nio.ch.Net.pollConnect(Native Method)
  18.     at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
  19.     at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
  20.     at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
  21.     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:586)
  22.     at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
  23.     at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1757)
  24.     at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1711)
  25.     at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:707)
  26. 2021-11-14 20:16:28,525 WARN hdfs.DataStreamer: Abandoning BP-2051009381-192.168.160.8-1635954925420:blk_1073742189_1366
  27. 2021-11-14 20:16:28,533 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[192.168.192.10:50010,DS-829fd615-4b31-4379-874a-ad06769d138e,DISK]
  28. 2021-11-14 20:16:29,557 WARN hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073742190_1367
  29. java.net.ConnectException: Connection refused
  30.     at java.base/sun.nio.ch.Net.pollConnect(Native Method)
  31.     at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
  32.     at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
  33.     at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
  34.     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:586)
  35.     at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
  36.     at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1757)
  37.     at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1711)
  38.     at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:707)
  39. 2021-11-14 20:16:29,557 WARN hdfs.DataStreamer: Abandoning BP-2051009381-192.168.160.8-1635954925420:blk_1073742190_1367
  40. 2021-11-14 20:16:29,569 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[192.168.192.11:50010,DS-3c3a744b-d53c-4cb5-97ac-4dd3e128f6a7,DISK]
  41. 2021-11-14 20:16:29,588 WARN hdfs.DataStreamer: DataStreamer Exception
  42. org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /kafka/last-test-5/14-11-21/sensor-data.1636917373340.tmp could only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
  43.     at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2219)
  44.     at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
  45.     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2789)
  46.     at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:892)
  47.     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:574)
  48.     at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  49.     at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
  50.     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
  51.     at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999)
  52.     at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927)
  53.     at java.security.AccessController.doPrivileged(Native Method)
  54.     at javax.security.auth.Subject.doAs(Subject.java:422)
  55.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
  56.     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
  58.     at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1573)
  59.     at org.apache.hadoop.ipc.Client.call(Client.java:1519)
  60.     at org.apache.hadoop.ipc.Client.call(Client.java:1416)
  61.     at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
  62.     at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129)
  63.     at jdk.proxy2/jdk.proxy2.$Proxy14.addBlock(Unknown Source)
  64.     at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:530)
  65.     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  66.     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  67.     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  68.     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  69.     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
  70.     at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
  71.     at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
  72.     at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
  73.     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
  74.     at jdk.proxy2/jdk.proxy2.$Proxy15.addBlock(Unknown Source)
  75.     at org.apache.hadoop.hdfs.DFSOutputStream.addBlock(DFSOutputStream.java:1084)
  76.     at org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1898)
  77.     at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1700)
  78.     at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:707)
  79. 2021-11-14 20:16:29,590 WARN hdfs.DFSClient: Error while syncing

数据写入HDFS，但生成的文件大小为0字节，并且文件中没有可用内容。
有人知道是什么原因导致这个错误以及如何修复它吗？
用于此项目的Docker图像：

  • bde 2020（适用于Hadoop）
  • （为Kafka和Zookeeper）

为了重现这个问题，我创建了一个git repo，您可以在其中提取项目来重新创建错误https://github.com/Benjaminbakir/Big-data-test
您还必须在本地计算机上安装Flume以运行agent.conf文件。
可以使用以下命令运行该文件（您必须使用cd命令转到存储配置文件的目录）：flume-ng代理-c . -f代理.conf --名称代理-Xmx 512 m
最后，您需要将以下内容添加到etc/hosts文件中：

  • 127.0.0.1本地主机名节点数据节点1数据节点2数据节点3
  • ：：1本地主机名节点数据节点1数据节点2数据节点3

现在，当您使用Kafka生成器向名为“test”的主题发送消息时，应该会出现错误。
创建Kafka主题的命令：/opt/bitnami/Kafka/bin/kafka-topics.sh--创建--引导服务器本地主机：9092 --主题测试--分区3 --复制因子1
用于创建生成方的命令：-kafka-console-producer.sh------------------------------------------------------------
如果有什么不清楚的地方，请告诉我，我会尽量解释得更详细。
PS：Hadoop集群运行正常，datanodes和namenode正在运行，用户可以通过Hadoop Web UI手动下载/上传文件，但当通过Kafka & Flume发送数据时，会出现此错误。

hadoop

1条答案

按热度按时间
dgjrabp2

dgjrabp21#

你只需要做一些“端口代理”，比如把ip：port转发到other_ip：port

赞(0）分享  回复(0）举报  2022-12-11
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com