如何在flume中设置日志文件名

6xfqseft  于 2021-06-04  发布在  Flume
关注(0)|答案(2)|浏览(539)

我正在使用apacheflume进行日志收集。这是我的配置文件

httpagent.sources = http-source
httpagent.sinks = local-file-sink
httpagent.channels = ch3

# Define source properties

httpagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
httpagent.sources.http-source.channels = ch3
httpagent.sources.http-source.port = 8082

# Local File Sink

httpagent.sinks.local-file-sink.type = file_roll
httpagent.sinks.local-file-sink.channel = ch3
httpagent.sinks.local-file-sink.sink.directory = /home/avinash/log_dir
httpagent.sinks.local-file-sink.sink.rollInterval = 21600

# Channels

httpagent.channels.ch3.type = memory
httpagent.channels.ch3.capacity = 1000

我的应用程序运行得很好,我的问题是在logdir中,文件默认使用了一些随机数(我猜是它的时间戳)时间戳。
如何为日志文件提供正确的文件名后缀?

olmpazwi

olmpazwi1#

对于源代码,您可以根据shell脚本执行命令来跟踪、预挂起或附加详细信息。下面是一个示例:


# Describe/configure the source for tailing file

 httpagent.sources.source.type = exec
 httpagent.sources.source.shell = /bin/bash -c
 httpagent.sources.source.command = tail -F /path/logs/*_details.log
 httpagent.sources.source.restart = true
 httpagent.sources.source.restartThrottle = 1000
 httpagent.sources.source.logStdErr = true
sczxawaw

sczxawaw2#

看看文档,似乎没有参数来配置要创建的文件的名称。我已经找到了一些隐藏参数的源代码,但是没有人:)
深入到实现的细节,文件名似乎是由 PathManager 班级:

private PathManager pathController;
...
@Override
public Status process() throws EventDeliveryException {
    ...
    if (outputStream == null) {
        File currentFile = pathController.getCurrentFile();
        logger.debug("Opening output stream for file {}", currentFile);
        try {
            outputStream = new BufferedOutputStream(new FileOutputStream(currentFile));
    ...
}

正如您已经注意到的,它基于当前时间戳(显示构造函数和下一个文件getter):

public PathManager() {
    seriesTimestamp = System.currentTimeMillis();
    fileIndex = new AtomicInteger();
}

public File nextFile() {
    currentFile = new File(baseDirectory, seriesTimestamp + "-" + fileIndex.incrementAndGet());
    return currentFile;
}

所以,我认为你唯一的可能就是扩展文件滚动槽并覆盖 process() 方法以使用自定义路径控制器。

相关问题