我试着用hdfsFlume运行Flume。hdfs正确地运行在不同的机器上,我甚至可以从flume机器与hdfs交互,但是当我运行flume并向它发送事件时,我得到以下错误:
2013-05-26 14:22:11,399 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:456)] HDFS IO error
java.io.IOException: Callable timed out after 25000 ms
at org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:352)
at org.apache.flume.sink.hdfs.HDFSEventSink.append(HDFSEventSink.java:727)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:430)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)
at java.util.concurrent.FutureTask.get(FutureTask.java:119)
at org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
... 5 more
同样,连接性不是问题,因为我可以使用hadoop命令行与hdfs交互(flume机器不是datanode)。最奇怪的是,在杀死flume之后,我可以看到tmp文件是在hdfs中创建的,但是它是空的(并且.tmp扩展名仍然存在)。
你知道为什么会这样吗?谢谢!
1条答案
按热度按时间kfgdxczn1#
检查三件事,如果你的防火墙是关闭的,即iptables应该停止。第二,财产的价值
agent.sinks.hdfs-sink.hdfs.path = hdfs://PUBLIC_IP:8020/user/hdfs/flume
而不是私有ip。和改变agent.sinks.hdfs-sink.hdfs.callTimeout = 180000
因为默认值是10000 ms,这对于hdfs来说React的时间非常少。谢谢,希尔帕