使用flume将数据从kafka摄取到hdfs::configurationexception:必须指定引导服务器

gdrx4gfi  于 2021-06-01  发布在  Hadoop
关注(0)|答案(1)|浏览(263)

我正在尝试使用flume从kafka源摄取数据到hdfs。下面是我的flume conf文件。

flume1.sources = kafka-source-1
flume1.channels = hdfs-channel-1
flume1.sinks = hdfs-sink-1
flume1.sources.kafka-source-1.type = org.apache.flume.source.kafka.KafkaSource
flume1.sources.kafka-source-1.bootstrap.servers = localhost:9092
flume1.sources.kafka-source-1.zookeeperConnect = localhost:2181
flume1.sources.kafka-source-1.topic = MYNEWSFEEDS
flume1.sources.kafka-source-1.batchSize = 100
flume1.sources.kafka-source-1.channels = hdfs-channel-1
flume1.channels.hdfs-channel-1.type   = memory
flume1.sinks.hdfs-sink-1.channel = hdfs-channel-1
flume1.sinks.hdfs-sink-1.type = hdfs
flume1.sinks.hdfs-sink-1.hdfs.writeFormat = Text
flume1.sinks.hdfs-sink-1.hdfs.fileType = DataStream
flume1.sinks.hdfs-sink-1.hdfs.filePrefix = test-events
flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true
flume1.sinks.hdfs-sink-1.hdfs.path = hdfs://quickstart.cloudera:8020/tmp
flume1.sinks.hdfs-sink-1.hdfs.rollCount=100
flume1.sinks.hdfs-sink-1.hdfs.rollSize=0
flume1.channels.hdfs-channel-1.capacity = 10000
flume1.channels.hdfs-channel-1.transactionCapacity = 1000

我正在使用以下命令运行flume代理:

sudo flume-ng agent --name flume1 --conf-file '/etc/flume-ng/conf/flafka.conf' Dflume.root.logger=TRACE,console

但我得到以下错误:
18/03/12 16:49:18 error node.abstractconfigurationprovider:source kafka-source-1由于在配置org.apache.flume.conf.configurationexception:必须在org.apache.flume.source.kafka.kafkasource.doconfigure(kafkasource)中指定引导服务器。java:330)在org.apache.flume.source.basicsourcesemantics.configure(基本资源管理)。java:65)在org.apache.flume.source.abstractpollablesource.configure(abstractpollablesource。java:63)在org.apache.flume.conf.configurables.configurable(configurables。java:41)在org.apache.flume.node.abstractconfigurationprovider.loadsources(abstractconfigurationprovider)。java:326)在org.apache.flume.node.abstractconfigurationprovider.getconfiguration(abstractconfigurationprovider)上。java:97)在org.apache.flume.node.pollingpropertiesfileconfigurationprovider$filewatcherrunnable.run(pollingpropertiesfileconfigurationprovider)。java:140)在java.util.concurrent.executors$runnableadapter.call(executors。java:471)在java.util.concurrent.futuretask.runandreset(futuretask。java:304)在java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.access$301(scheduledthreadpoolexecutor。java:178)在java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.run(scheduledthreadpoolexecutor。java:293)位于java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor。java:1145)在java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor。java:615)在java.lang.thread.run(线程。java:745)
虽然我已经在conf文件中指定了引导服务器,但是它仍然给出了相同的错误。尝试了许多排列和组合,但没有成功。

klh5stk1

klh5stk11#

根据官方的javadoc,你应该

flume1.sources.kafka-source-1.bootstrap.servers = localhost:9092

具有

flume1.sources.kafka-source-1.kafka.bootstrap.servers = localhost:9092

相关问题