“事件大小大于指定的事件大小:2500考虑增加最大事件大小“

ego6inou  于 2021-06-04  发布在  Flume
关注(0)|答案(1)|浏览(515)

我正在尝试使用 org.apache.flume.source.MultiportSyslogTCPSource . 设置和配置成功,但在flume从syslog生成事件时出现以下错误。

[INFO ] [2019-03-28 13:22:27.217] [[channel=file-channel] - CheckpointBackUpThread] [org.apache.flume.channel.file.EventQueueBackingStoreFile] - Checkpoint backup completed.

[WARN ] [2019-03-28 13:22:31.853] [NioProcessor-2] 

[org.apache.flume.source.MultiportSyslogTCPSource] - Event size larger than specified event size: 2500. Consider increasing the max event size.
    [INFO ] [2019-03-28 13:22:35.686] [SinkRunner-PollingRunner-DefaultSinkProcessor] [org.apache.flume.sink.LoggerSink] - Event: { headers:{flume.syslog.status=Invalid} body: 31 33 33 62 32 20 74 3C 31 33 34 3E 4D 61 72 20 133b2 t<134>Mar  }

我的配置如下:


# source,channel and sink

testagent.sources = testlog
testagent.channels = file-channel
testagent.sinks = logger-sink

# source

testagent.sources.testlog.type = org.apache.flume.source.MultiportSyslogTCPSource
testagent.sources.testlog.ports = 9002
testagent.sources.testlog.host = 127.0.0.1

# sink

testagent.sinks.logger-sink.type = logger

# channel

testagent.channels.file-channel.type = file
testagent.channels.file-channel.dataDirs = /test/data/01/
testagent.channels.file-channel.checkpointDir = /test/data/01/checkpoint
testagent.channels.file-channel.useDualCheckpoints = true
testagent.channels.file-channel.backupCheckpointDir = /test/data/01/checkpoint-backup
testagent.channels.file-channel.transactionCapacity = 10000
testagent.channels.file-channel.checkpointInterval = 20000
testagent.channels.file-channel.maxFileSize = 1072692224
testagent.channels.file-channel.minimumRequiredSpace = 524288000
testagent.channels.file-channel.capacity = 1000000
testagent.channels.file-channel.keep-alive = 3
testagent.channels.file-channel.checkpointOnClose = true
testagent.sinks.logger-sink.channel = file-channel
testagent.sources.testlog.channels  = file-channel

从我的测试应用程序生成的日志应该成功地显示在flume代理日志中,并且应该从每个日志语句(行)生成事件。

b09cbbtk

b09cbbtk1#

更新配置中事件的大小,如下所示:

testagent.sinks.logger-sink.maxBytesToLog = 256

事件的默认大小为16字节。

相关问题