flafka:kafka到flume代理hdfs不接收

kyxcudwk  于 2021-06-02  发布在  Hadoop
关注(0)|答案(0)|浏览(201)

我正在创建一个flume代理,该代理接收来自kafka主题的数据,通道是kafka主题,接收器是hdfs。代理应该从kafka主题获取数据,并将其放到hdfs中的daily文件夹中。我在cloudera发行。它当前没有接收数据。未给出错误消息,数据正在成功填充主题。

sandbox.sources = kafka
sandbox.channels= channel
sandbox.sinks=sink_to_hdfs

# source configuration

sandbox.sources.kafka.type = org.apache.flume.source.kafka.KafkaSource
sandbox.sources.kafka.zookeeperConnect = zookeeper_address:2181/kafka
sandbox.sources.kafka.topic= topic
sandbox.sources.kafka.groupId= flume
sandbox.sources.kafka.channels=channel

# channel configuration

sandbox.channels.channel.type=org.apache.flume.channel.kafka.KafkaChannel
sandbox.channels.channel.brokerList=kafkabroker_server:9092 
sandbox.channels.channel.topic=kafka-channel-01
sandbox.channels.channel.zookeeperConnect = zookeeper_address:2181/kafka

# sink configuration

sandbox.sinks.sink_to_hdfs.channel= channel
sandbox.sinks.sink_to_hdfs.type=hdfs
sandbox.sinks.sink_to_hdfs.writeFormat=Text
sandbox.sinks.sink_to_hdfs.hdfs.fileType= DataStream
sandbox.sinks.sink_to_hdfs.hdfs.path=/user/myname/flume/%{topic}/%Y-%m-%d

sandbox.sinks.sink_to_hdfs.hdfs.rollInterval=10 
sandbox.sinks.sink_to_hdfs.hdfs.rollSize=8000000
sandbox.sinks.sink_to_hdfs.hdfs.rollCount=0
sandbox.sinks.sink_to_hdfs.hdfs.idleTimeout=3600

# channel capacity

sandbox.channels.channel.transactionCapacity = 1000
sandbox.channels.channel.capacity= 10000

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题