无法在flume中存储twitter数据

fruv7luv  于 2021-06-04  发布在  Flume
关注(0)|答案(1)|浏览(408)

我们成功地从twitter中提取了数据,但是我们无法使用flume将其保存到我们的系统中。您能解释一下吗

x8goxv8g

x8goxv8g1#

you might have problem in channel or sink may be that's why u r data is not storing in hdfs.

try to understan  this one

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://yourIP:8020/user/flume/tweets/%Y/%M/%d/%H/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

and chek with jps if your data node and namenode is working

相关问题