Flume和/或log4j有问题。我有javaee应用程序,log4j和flume appender=一切正常。
当我用下面的配置运行flume时,一切都很好。
这里是flume conf文件
# define components of agent
a1.sources = avroSource
a1.channels = ch1
a1.sinks = hdfsSink
# define source
a1.sources.avroSource.type = avro
a1.sources.avroSource.bind = localhost
a1.sources.avroSource.port = 41415
a1.sources.avroSource.channels = ch1
# define interceptors
a1.sources.avroSource.interceptors = i1 i2 i3 i4 i5
a1.sources.avroSource.interceptors.i1.type = timestamp
a1.sources.avroSource.interceptors.i2.type = host
a1.sources.avroSource.interceptors.i3.type = regex_filter
a1.sources.avroSource.interceptors.i4.type = regex_filter
a1.sources.avroSource.interceptors.i5.type = static
a1.sources.avroSource.interceptors.i2.useIP = false
a1.sources.avroSource.interceptors.i2.hostHeader = host
a1.sources.avroSource.interceptors.i3.regex = .*AppLogging.*
a1.sources.avroSource.interceptors.i3.excludeEvents = false
a1.sources.avroSource.interceptors.i4.regex = .*;ERORR;.*
a1.sources.avroSource.interceptors.i4.excludeEvents = true
a1.sources.avroSource.interceptors.i5.key = instance
a1.sources.avroSource.interceptors.i5.value = JBT_TEST
# define channel
a1.channels.ch1.type = memory
a1.channels.ch1.capacity = 1000
a1.channels.ch1.transactionCapacity = 100
# define sink
a1.sinks.hdfsSink.type = hdfs
a1.sinks.hdfsSink.channel = ch1
a1.sinks.hdfsSink.hdfs.writeFormat = Text
a1.sinks.hdfsSink.hdfs.filePrefix = %{host}
a1.sinks.hdfsSink.hdfs.fileType = DataStream
a1.sinks.hdfsSink.hdfs.path = hdfs://192.168.79.128:9000/hdfs/mr/input/flume/%{instance}/
a1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
a1.sinks.hdfsSink.hdfs.rollInterval = 0
a1.sinks.hdfsSink.hdfs.rollCount = 0
a1.sinks.hdfsSink.hdfs.rollSize = 0
a1.sinks.hdfsSink.hdfs.batchSize = 1
如果我将Flume从hdfs改为file\u roll,flume代理将创建日志文件,但不包含任何内容。这里是flume的新conf文件。
# define components of agent
a1.sources = avroSource
a1.channels = ch1
a1.sinks = file
# define source
a1.sources.avroSource.type = avro
a1.sources.avroSource.bind = localhost
a1.sources.avroSource.port = 41415
a1.sources.avroSource.channels = ch1
# define channel
a1.channels.ch1.type = memory
a1.channels.ch1.capacity = 1000
a1.channels.ch1.transactionCapacity = 100
# define sink
a1.sinks.file.type = file_roll
a1.sinks.file.channel = ch1
a1.sinks.file.sink.directory = c:/temp/
a1.sinks.file.sink.rollInterval = 600
我犯了什么错?
1条答案
按热度按时间kxxlusnw1#
在配置末尾添加以下两行,以将源和接收器与通道连接起来。让我知道这是否有效。