尝试通过flume提取twitter数据时出错

omtl5h9j  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(274)

我使用flume下载twitter数据,但是当我运行flume命令时,我得到了这个错误。请帮忙。

2016-08-13 17:21:12,945 (conf-file-poller-0) [ERROR - org.apache.flume.node.Poll ingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertie FileConfigurationProvider.java:149)] Unhandled error java.lang.AssertionError: java.lang.reflect.InvocationTargetException at twitter4j.HttpClientFactory.getInstance(HttpClientFactory.java:81) at twitter4j.TwitterStreamImpl.<init>(TwitterStreamImpl.java:51) at twitter4j.TwitterStreamFactory.<clinit>(TwitterStreamFactory.java:40) at org.apache.flume.source.twitter.TwitterSource.configure(TwitterSource .java:115) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSources(Abstr actConfigurationProvider.java:326) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration( AbstractConfigurationProvider.java:97) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$File WatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51 1) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask. run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct orAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC onstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:408) at twitter4j.HttpClientFactory.getInstance(HttpClientFactory.java:73) ... 14 more Caused by: java.lang.NoSuchMethodError: twitter4j.conf.Configuration.getHttpClie ntConfiguration()Ltwitter4j/HttpClientConfiguration; at twitter4j.StreamingReadTimeoutConfiguration.isGZIPEnabled(TwitterStre amImpl.java:804) at twitter4j.HttpClientBase.<init>(HttpClientBase.java:25) at twitter4j.HttpClientImpl.<init>(HttpClientImpl.java:55) ... 19 more

配置文件如下:-

TwitterAgent.sources = Twitter
    TwitterAgent.channels = MemChannel
    TwitterAgent.sinks = HDFS

    TwitterAgent.sources.Twitter.type =    org.apache.flume.source.twitter.TwitterSource
    TwitterAgent.sources.Twitter.channels = MemChannel
    TwitterAgent.sources.Twitter.consumerKey = FMCaWqdvqJPzzmY76uPAzJphX
    TwitterAgent.sources.Twitter.consumerSecret =  me1DRFssnyirpj3j1cOC4rByvxTR1F2LfF2n3udaOXb6K9yWiZ
    TwitterAgent.sources.Twitter.accessToken = 753216791690653696-         CN3n8Krf8DBI5KorjnfjlmoVCFf7EnE
    TwitterAgent.sources.Twitter.accessTokenSecret = KIEEPE6XDthMZ6peCoVYNDXKOy1ElTbF7hjXFggW7NHQK

    TwitterAgent.sources.Twitter.keywords = india , elections , congress

    TwitterAgent.sinks.HDFS.channel = MemChannel
    TwitterAgent.sinks.HDFS.type = hdfs
    TwitterAgent.sinks.HDFS.hdfs.path =hdfs://localhost:9000/user/data/tweets_raw/
    TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
    TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
    TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
    TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
    TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

    TwitterAgent.channels.MemChannel.type = memory
    TwitterAgent.channels.MemChannel.capacity = 10000
    TwitterAgent.channels.MemChannel.transactionCapacity = 100

Flume命令是:-

bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -   Dflume.root.logger=DEBUG,console -n TwitterAgent

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题