flume配置文件

k0pti3hp  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(421)

Flume和/或log4j有问题。我有javaee应用程序,log4j和flume appender=一切正常。
当我用下面的配置运行flume时,一切都很好。
这里是flume conf文件


# define components of agent

a1.sources = avroSource
a1.channels = ch1
a1.sinks = hdfsSink

# define source

a1.sources.avroSource.type = avro
a1.sources.avroSource.bind = localhost
a1.sources.avroSource.port = 41415
a1.sources.avroSource.channels = ch1

# define interceptors

a1.sources.avroSource.interceptors = i1 i2 i3 i4 i5
a1.sources.avroSource.interceptors.i1.type = timestamp
a1.sources.avroSource.interceptors.i2.type = host
a1.sources.avroSource.interceptors.i3.type = regex_filter
a1.sources.avroSource.interceptors.i4.type = regex_filter
a1.sources.avroSource.interceptors.i5.type = static
a1.sources.avroSource.interceptors.i2.useIP = false
a1.sources.avroSource.interceptors.i2.hostHeader = host
a1.sources.avroSource.interceptors.i3.regex = .*AppLogging.*
a1.sources.avroSource.interceptors.i3.excludeEvents = false
a1.sources.avroSource.interceptors.i4.regex = .*;ERORR;.*
a1.sources.avroSource.interceptors.i4.excludeEvents = true
a1.sources.avroSource.interceptors.i5.key = instance
a1.sources.avroSource.interceptors.i5.value = JBT_TEST

# define channel

a1.channels.ch1.type = memory
a1.channels.ch1.capacity = 1000
a1.channels.ch1.transactionCapacity = 100

# define sink

a1.sinks.hdfsSink.type = hdfs
a1.sinks.hdfsSink.channel = ch1
a1.sinks.hdfsSink.hdfs.writeFormat = Text
a1.sinks.hdfsSink.hdfs.filePrefix =  %{host}
a1.sinks.hdfsSink.hdfs.fileType = DataStream
a1.sinks.hdfsSink.hdfs.path = hdfs://192.168.79.128:9000/hdfs/mr/input/flume/%{instance}/
a1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
a1.sinks.hdfsSink.hdfs.rollInterval = 0
a1.sinks.hdfsSink.hdfs.rollCount = 0
a1.sinks.hdfsSink.hdfs.rollSize = 0
a1.sinks.hdfsSink.hdfs.batchSize = 1

如果我将Flume从hdfs改为file\u roll,flume代理将创建日志文件,但不包含任何内容。这里是flume的新conf文件。


# define components of agent

a1.sources = avroSource
a1.channels = ch1
a1.sinks = file

# define source

a1.sources.avroSource.type = avro
a1.sources.avroSource.bind = localhost
a1.sources.avroSource.port = 41415
a1.sources.avroSource.channels = ch1

# define channel

a1.channels.ch1.type = memory
a1.channels.ch1.capacity = 1000
a1.channels.ch1.transactionCapacity = 100

# define sink

a1.sinks.file.type = file_roll
a1.sinks.file.channel = ch1
a1.sinks.file.sink.directory = c:/temp/
a1.sinks.file.sink.rollInterval = 600

我犯了什么错?

kxxlusnw

kxxlusnw1#

在配置末尾添加以下两行,以将源和接收器与通道连接起来。让我知道这是否有效。

a1.sources.avroSource.channels = ch1
a1.sinks.file.channel = ch1

相关问题