我正在创建一个flume代理,该代理接收来自kafka主题的数据,通道是kafka主题,接收器是hdfs。代理应该从kafka主题获取数据,并将其放到hdfs中的daily文件夹中。我在cloudera发行。它当前没有接收数据。未给出错误消息,数据正在成功填充主题。
sandbox.sources = kafka
sandbox.channels= channel
sandbox.sinks=sink_to_hdfs
# source configuration
sandbox.sources.kafka.type = org.apache.flume.source.kafka.KafkaSource
sandbox.sources.kafka.zookeeperConnect = zookeeper_address:2181/kafka
sandbox.sources.kafka.topic= topic
sandbox.sources.kafka.groupId= flume
sandbox.sources.kafka.channels=channel
# channel configuration
sandbox.channels.channel.type=org.apache.flume.channel.kafka.KafkaChannel
sandbox.channels.channel.brokerList=kafkabroker_server:9092
sandbox.channels.channel.topic=kafka-channel-01
sandbox.channels.channel.zookeeperConnect = zookeeper_address:2181/kafka
# sink configuration
sandbox.sinks.sink_to_hdfs.channel= channel
sandbox.sinks.sink_to_hdfs.type=hdfs
sandbox.sinks.sink_to_hdfs.writeFormat=Text
sandbox.sinks.sink_to_hdfs.hdfs.fileType= DataStream
sandbox.sinks.sink_to_hdfs.hdfs.path=/user/myname/flume/%{topic}/%Y-%m-%d
sandbox.sinks.sink_to_hdfs.hdfs.rollInterval=10
sandbox.sinks.sink_to_hdfs.hdfs.rollSize=8000000
sandbox.sinks.sink_to_hdfs.hdfs.rollCount=0
sandbox.sinks.sink_to_hdfs.hdfs.idleTimeout=3600
# channel capacity
sandbox.channels.channel.transactionCapacity = 1000
sandbox.channels.channel.capacity= 10000
暂无答案!
目前还没有任何答案,快来回答吧!