flume代理不包含任何有效通道

fykwrbwg  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(654)

我不熟悉Flume。我试图从twitter上获取数据,但我没有成功(我正在使用cloudera快速启动)
我的conf文件如下所示:

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel

我已经添加了来自twitter帐户consumerkey、consumersecret、accesstoken、accesstokensecret、keywords和path的所有值

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.fileType =  DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollsize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

我用来执行conf文件的命令是:

flume-ng agent --conf conf --conf-file flume.conf -Dflume.root.logger=DEBUG,console -name TwitterAgent

我得到的错误是:

18/06/27 12:17:18 WARN conf.FlumeConfiguration: Agent configuration for 'TwitterAgent' does not contain any valid channels. Marking it as invalid.
18/06/27 12:17:18 WARN conf.FlumeConfiguration: Agent configuration invalid for agent 'TwitterAgent'. It will be removed.
18/06/27 12:17:18 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: []
18/06/27 12:17:18 WARN node.AbstractConfigurationProvider: No configuration found for this host:TwitterAgent
18/06/27 12:17:18 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

请告诉我。

2w3rbyxf

2w3rbyxf1#

我认为您的执行命令有问题,错误在于查找配置文件。
命令应该是

flume-ng agent -c conf -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent

必须指定配置文件路径。您可以尝试使用-f conf/flume.conf而不是-f flume.conf

相关问题