使用tshark和flume捕获网络流量

wfsdck30 于 2021-06-04 发布在 Flume

关注(0)|答案(1)|浏览(412)

大家好，我正在尝试使用tshark捕获网络流量，并使用apacheflume将这些结果发送给spark。
问题是，当我在flume的配置中使用exec源时，flume不工作，它在启动后立即停止
我的配置：

agent.sources = tsharkSource 
agent.channels = memoryChannel 
agentasinks = avroSink 

# Configuring the sources to pull the bashes from Tshark

agent.sources.tsharkSource.type = exec 
agent.sources.tsharkSource.command = tshark -T json 
agent.sources.tsharkSource.channels = memoryChannel 

# Configuring the sink to push logs to spark change hostname to 116's ip adress

agent.sinks.avroSink.type = avro 
agent.sinks.avroSink.channel = memoryChannel 
agent.sinks.avroSink.hostname = 192.168.1.112 
agent.sinks.avroSink.port= 6969 

# Configuring the memory channel

agent.channels.memoryChannel.type = memory 
agent.channels.memoryChannel.capacity = 1000 
agent.channels.memoryChannel.transactionCapacity = 100

shell输出：

flume-ng agent -f conf/flume-conf.properties -n agent 

Warning: No configuration directory set! Use --conf <dir> to override. 
Info: Including Hive libraries found via () for Hive access 
+ exec /usr/lib/jum/jaua-1.11.0-openjdk-amd64//bin/jaua -Xmx20m -cp 
1/home/oshiflume/flume/apache-flume-1.8.0-bin/lib/*:/libfle - 
Djava.library.path= org.apache.flume.node.Application -f conf/flume- 
conf.properties -n agent 
log4j:WARN No appenders could be found for logger (org.apache.flume.node.Application). 
log4j:WARN Please initialize the log4j system properly. 
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.htmlDnoconfig for more info.

flume flume-ng tshark

来源：https://stackoverflow.com/questions/51764313/capturing-networking-traffic-using-tshark-and-flume

1条答案

按热度按时间

p4rjhz4m1#

你的执行命令是

flume-ng agent -f conf/flume-conf.properties -n agent

我看到两个错误。首先，必须使用 -c conf flume配置文件通常命名为 some-config.conf 控制台上的警告是 No configuration directory set! Use --conf 、-c和--conf是同一件事。
您可能需要将配置文件从flume-conf.properties重命名为flume.conf
作为解决方案，您可以尝试以下命令：

flume-ng agent -c conf -f conf/flume.conf -n agent

如果要在执行后显示日志，请使用以下命令

flume-ng agent -c conf -f conf/flume.conf -n agent -Dflume.root.logger=INFO,console

要显示日志，log4j.properties必须在conf目录中作为conf/log4j.properties。
我的财产如下：

flume.root.logger=INFO,LOGFILE
flume.log.dir=./logs
flume.log.file=flume.log

log4j.logger.org.apache.flume.lifecycle = INFO
log4j.logger.org.jboss = WARN
log4j.logger.org.mortbay = INFO
log4j.logger.org.apache.avro.ipc.NettyTransceiver = WARN
log4j.logger.org.apache.hadoop = INFO
log4j.logger.org.apache.hadoop.hive = ERROR

# Define the root logger to the system property "flume.root.logger".

log4j.rootLogger=${flume.root.logger}

log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.LOGFILE.MaxFileSize=100MB
log4j.appender.LOGFILE.MaxBackupIndex=10
log4j.appender.LOGFILE.File=${flume.log.dir}/${flume.log.file}
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n

log4j.appender.DAILY=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.DAILY.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.DAILY.rollingPolicy.ActiveFileName=${flume.log.dir}/${flume.log.file}
log4j.appender.DAILY.rollingPolicy.FileNamePattern=${flume.log.dir}/${flume.log.file}.%d{yyyy-MM-dd}
log4j.appender.DAILY.layout=org.apache.log4j.PatternLayout
log4j.appender.DAILY.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n

赞(0）回复(0）举报 2021-06-04

我来回答

使用tshark和flume捕获网络流量

1条答案

相关问题

热门标签

最新问答