我的flume内存通道有问题。我运行了一个flume代理,它淹没了内存通道,日志开始溢出“通道已满,现在无法写入数据。源将在250毫秒后重试“
到现在为止,一直都还不错。我停止代理,编辑flume.conf以增加容量,然后重试。问题是,在启动时,flume已经溢出了相同的消息:
`16/05/14 00:21:48 INFO node.Application: Starting new configuration:
{ sourceRunners:{s1=EventDrivenSourceRunner: { source:Spool
Directory source s1: { spoolDir: /home/vagrant/logs } }}
sinkRunners:{kafka-avro-sink2=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@63203b59
counterGroup:{ name:null counters:{} } }, kafka-avro-
sink1=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@591882e6
counterGroup:{ name:null counters:{} } }} channels:
{mem1=org.apache.flume.channel.MemoryChannel{name: mem1}} }
16/05/14 00:21:48 INFO node.Application: Starting Channel mem1
16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup:
Monitored counter group for type: CHANNEL, name: mem1: Successfully
registered new MBean.
16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup:
Component type: CHANNEL, name: mem1 started
16/05/14 00:21:48 INFO node.Application: Starting Sink kafka-avro-
sink2
16/05/14 00:21:48 INFO node.Application: Starting Sink kafka-avro-sink1
16/05/14 00:21:48 INFO node.Application: Starting Source s1
16/05/14 00:21:48 INFO source.SpoolDirectorySource:
SpoolDirectorySource source starting with directory:
/home/vagrant/logs
16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup:
Monitored counter group for type: SOURCE, name: s1: Successfully
registered new MBean.
16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup:
Component type: SOURCE, name: s1 started
16/05/14 00:21:49 WARN source.SpoolDirectorySource: The channel is
full, and cannot write data now. The source will try again after
250 milliseconds
16/05/14 00:21:49 INFO avro.ReliableSpoolingFileEventReader: Last
read was never committed - resetting mark position.
16/05/14 00:21:49 WARN source.SpoolDirectorySource: The channel is
full, and cannot write data now. The source will try again after
500 milliseconds`
所以当我启动代理时,频道已经满了。如何手动重置或清除?
为了一个类似的问题,我找了好几个小时都没有找到。一个稍微恼人的问题是https://flume.apache.org/ 几乎所有与flume相关的内容都被google编入索引。因此,搜索“Flume频道总是满的”,例如,将始终给出https://flume.apache.org/ 作为第一个结果。
1条答案
按热度按时间erhoui1w1#
似乎无法为频道分配足够的内存。当事件的传入速率高于内存通道的可用内存时。
尝试使用下面的例子:xmx是一个示例选项,您可以根据并行运行的代理数逐渐增加内存分配的最佳值。
例如:
此外,您还需要通过引用flume指南中的这个内存通道来检查代理配置文件中的其他配置参数。