hadoop使用flume拉twitter推文

6ie5vjzr  于 2021-05-31  发布在  Hadoop
关注(0)|答案(0)|浏览(191)

当我尝试获取twitter tweets时,出现了一个接收器错误。我添加了twitterapi配置,并在hdfs中创建了一个目录。我不确定我做错了什么。我正在使用hadoop2.0.0-cdh4.2.1

ERROR flume.SinkRunner: Unable to deliver event. 
ERROR hdfs.HDFSEventSink: process failed

例外情况随之而来。

java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

Flume.conf


# Naming the components on the current agent.

TwitterAgent.sources = twitter
TwitterAgent.channels = memoryChannel
TwitterAgent.sinks = HDFS

# Describing/Configuring the source

TwitterAgent.sources.twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.twitter.consumerKey = Xxx
TwitterAgent.sources.twitter.consumerSecret = Xxx 
TwitterAgent.sources.twitter.accessToken = Xxx
TwitterAgent.sources.twitter.accessTokenSecret = Xxx
TwitterAgent.sources.twitter.maxBatchDurationMillis = 200 
TwitterAgent.sources.twitter.channels = memoryChannel
TwitterAgent.sources.twitter.keywords = lsu

TwitterAgent.channels.memoryChannel.type = memory
TwitterAgent.channels.memoryChannel.capacity = 10000
TwitterAgent.channels.memoryChannel.transactionCapacity = 1000

TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.channel = memoryChannel
TwitterAgent.sinks.HDFS.hdfs.path = hdfs:/user/flume/tweets/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.sinks.HDFS.hdfs.useLocalTimeStamp = true

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题