flume hdfs rollsize不适用于多通道和多Flume

1rhkuytd  于 2021-06-04  发布在  Flume
关注(0)|答案(1)|浏览(457)

我正在尝试使用flume ng获取128mb的日志信息,并将其放入hdfs中的一个文件中。但hdfs滚动选项不起作用。flume ng每秒发送日志文件。如何修复flume.conf文件?

agent01.sources = avroGenSrc
agent01.channels = memoryChannel hdfsChannel
agent01.sinks = fileSink hadoopSink

# For each one of the sources, the type is defined

agent01.sources.avroGenSrc.type = avro
agent01.sources.avroGenSrc.bind = dev-hadoop03.ncl
agent01.sources.avroGenSrc.port = 3333

# The channel can be defined as follows.

agent01.sources.avroGenSrc.channels = memoryChannel hdfsChannel

# Each sink's type must be defined

agent01.sinks.fileSink.type = file_roll
agent01.sinks.fileSink.sink.directory = /home1/irteam/flume/data
agent01.sinks.fileSink.sink.rollInterval = 3600
agent01.sinks.fileSink.sink.batchSize = 100

# Specify the channel the sink should use

agent01.sinks.fileSink.channel = memoryChannel

agent01.sinks.hadoopSink.type = hdfs
agent01.sinks.hadoopSink.hdfs.useLocalTimeStamp = true
agent01.sinks.hadoopSink.hdfs.path = hdfs://dev-hadoop04.ncl:9000/user/hive/warehouse/raw_logs/year=%Y/month=%m/day=%d
agent01.sinks.hadoopSink.hdfs.filePrefix = AccessLog.%Y-%m-%d.%Hh
agent01.sinks.hadoopSink.hdfs.fileType = DataStream
agent01.sinks.hadoopSink.hdfs.writeFormat = Text
agent01.sinks.hadoopSink.hdfs.rollInterval = 0
agent01.sinks.hadoopSink.hdfs.rollSize = 134217728
agent01.sinks.hadoopSink.hdfs.rollCount = 0

# Specify the channel the sink should use

agent01.sinks.hadoopSink.channel = hdfsChannel

# Each channel's type is defined.

agent01.channels.memoryChannel.type = memory
agent01.channels.hdfsChannel.type = memory

# Other config values specific to each type of channel(sink or source)

# can be defined as well

# In this case, it specifies the capacity of the memory channel

agent01.channels.memoryChannel.capacity = 100000
agent01.channels.memoryChannel.transactionCapacity = 10000

agent01.channels.hdfsChannel.capacity = 100000
agent01.channels.hdfsChannel.transactionCapacity = 10000
vddsk6oq

vddsk6oq1#

我找到了这个解决办法。dfs.replication不匹配导致此问题。
在我的hadoop配置中( hadoop-2.7.2/etc/hadoop/hdfs-site.xml )

<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>

我有2个数据节点,所以我把它改成

<property>
  <name>dfs.replication</name>
  <value>2</value>
</property>

我加了配置 flume.conf ```
agent01.sinks.hadoopSink.hdfs.minBlockReplicas = 2

谢谢你的帮助
https://qnalist.com/questions/5015704/hit-max-consecutive-under-replication-rotations-error
和
FlumehdfsFlume不断滚动小文件

相关问题